Docker way

Systemd way

The datarhei Core is a process management solution for FFmpeg that offers a range of interfaces for media content, including HTTP, RTMP, SRT, and storage options. It is optimized for use in virtual environments such as Docker. It has been implemented in various contexts, from small-scale applications like Restreamer to large-scale, multi-instance frameworks spanning multiple locations, such as dedicated servers, cloud instances, and single-board computers. The datarhei Core stands out from traditional media servers by emphasizing FFmpeg and its capabilities rather than focusing on media conversion.

__

Objectives of development

The objectives of development are:

• Unhindered use of FFmpeg processes

• Portability of FFmpeg, including management across development and production environments

• Scalability of FFmpeg-based applications through the ability to offload processes to additional instances

• Streamlining of media product development by focusing on features and design.

What issues have been resolved thus far?

Process management

• Run multiple processes via API

• Unrestricted FFmpeg commands in process configuration.

• Error detection and recovery (e.g., FFmpeg stalls, dumps)

• Referencing for process chaining (pipelines)

• Placeholders for storage, RTMP, and SRT usage (automatic credentials management and URL resolution)

• Logs (access to current stdout/stderr)

• Log history (configurable log history, e.g., for error analysis)

• Resource limitation (max. CPU and MEMORY usage per process)

• Statistics (like FFmpeg progress per input and output, CPU and MEMORY, state, uptime)

• Input verification (like FFprobe)

• Metadata (option to store additional information like a title)

Media delivery

• Configurable file systems (in-memory, disk-mount, S3)

• HTTP/S, RTMP/S, and SRT services, including Let's Encrypt

• Bandwidth and session limiting for HLS/MPEG DASH sessions (protects restreams from congestion)

• Viewer session API and logging

Misc

• HTTP REST and GraphQL API

• Swagger documentation

• Metrics incl. Prometheus support (also detects POSIX and cgroups resources)

• Docker images for fast setup of development environments up to the integration of cloud resources

Quick start

Changelog

Releases

The profiling endpoint allows you to fetch profiling information from a running datarhei Core instance. It has to be enabled with debug.profiling in the config.

Navigate you browser to /profiling where you can access different diagnostic solutions as described in https://go.dev/doc/diagnostics.

Settings for the database of operational data, e.g. processes, metadata, persisted session summaries, and so on.

Configuration

dir (string)

Directory for holding the operational data. The path is relative to where the binary is executed and the directory must exist.

The default value is ./config

The datarhei Core includes a simple RTMP server for publishing and playing streams. It is not enabled by default. You have to enable it in the config in the rtmp section or via the corresponding environemnt variables.

Example:

rtmp://127.0.0.1:1935/live/12345.stream

In the above example /live is the RTMP app and 12345.stream is the name of the resource.

Token

In order to protect access to the RTMP server you should define a token in the configuration. Only with a valid token it is possible to publish or play resources. The token has to be appended to the RTMP URL as a query string, e.g. rtmp://127.0.0.1:1935/live/12345.stream?token=abc.

As of version 16.12.0 you can write the token as last part of the path in URL instead of as a query string. The above example will looke like rtmp://127.0.0.1:1935/live/12345.stream/abc. This allows you to enter the token into the "stream key" field in some clients.

API

Via the RTMP endpoint in the API you can gather a list of the currently publishing RTMP resources.

The debugging settings can help to find and solve issues with the datarhei Core.

Configuration

{
   "debug": {
      "profiling": false,
      "force_gc": 0,
      "memory_limit_mbytes": 0,
   }
}

CORE_DEBUG_PROFILING=false
CORE_DEBUG_FORCE_GC=0
CORE_DEBUG_MEMORY_LIMIT_MBYTES=0

profiling (bool)

By setting this to true, the endpoint /profiling will be established where you can access different diagnostic solutions as described in https://go.dev/doc/diagnostics.

By default this setting is set to false.

force_gc (integer)

Golang is usually quite greedy when it comes to claim memory to itself. This settings lets you define the number of seconds between forcing the garbage collector to run in order return memory to the OS. If this is not set, the runtime will decide on its own when to run the garbage collector.

Alternatively, you can set the environment variable GOMEMLIMIT to a value in bytes in order to set a soft memory limit. This will influence the garbage colltector when the consumed memory comes close to this limit. If you use the GOMEMLIMIT environment variable you are advised to leave the force_gc option disabled.

The default for this setting is 0 (i.e. disabled).

memory_limit_mbytes (integer)

As of version 16.12.0 use this valuess to impose a soft limit to the memory consumption of the Core application itselft (i.e. with out the memory consumption of the ffmpeg processes). This has the same effect as setting the GOMEMLIMIT environment variable.

The provided value is the number of megabytes the Core application is allowed to consumed.

The default for this setting is 0 (i.e. no limit).

The /ping endpoint returns a plain text pong response. This can be used for liveliness and/or latency checks.

curl http://127.0.0.1:8080/ping

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_ping = client.ping()
print(core_ping)

The datarhei Core provides two filesystem abstractions that you can use in your FFmpeg process configurations.

Disk

The disk filesystem is the directory you defined in the configuration at storage.disk.dir. Any FFmpeg command will be restricted to this directory (or its subdirectories) for file access (read or write). One exception is /dev in order to access, e.g. USB cameras.

In a process configuration you can use the {diskfs} placeholder, such that you don't need to remember and write the configured path.

The contents of the disk filesystem are accessible read-only via the / path of the datarhei core HTTP server.

In order to access and modify the contents of the filesystem programmatically, you can use the corresponding API endpoints.

Memory

The datarhei Core has built-in memory filesystem. It is enabled by default and it is only accessible via HTTP. Its contents can be accessed via the /memfs path of the datarhei Core HTTP server.

In the configuration in the storage.memory section you can define different aspects of the memory file system, such as the maximum size of the filesystem, the maximum file size, password protection, and so on.

In a process configuration you can use the {memfs} placeholder, such that you don't need to remember and write the whole base URL.

The /memfs path is not read-only. Write access is protected via HTTP BasicAuth.

In order to access and modify the contents of the filesystem programmatically, you can use the corresponding API endpoints.

S3

The datarhei Core allows you to mount a S3 compatible filesystem. It is only accessible via HTTP. Its contents can be accessed via the configured path of the datarhei Core HTTP server.

In the configuration in the storage.s3 section you can define different aspects of the S3 filesystem, such as the login credentials, bucket name, and so on.

In a process configuration you can use the {fs:[name]} placeholder, such that you don't need to remember and write the whole base URL, where [name] is the configured name of the S3 filesystem, e.g. {fs:aws}.

The S3 filesystem HTTP mountpoint is not read-only. Write access is protected via HTTP BasicAuth.

In order to access and modify the contents of the filesystem programmatically, you can use the corresponding API endpoints.

Settings for collecting metrics of the core and FFmpeg processes.

Configuration

{
   "metrics": {
      "enable": false,
      "enable_prometheus": false,
      "range_sec": 300,
      "interval_sec": 2
   }
}

CORE_METRICS_ENABLE=false
CORE_METRICS_ENABLE_PROMETHEUS=false
CORE_METRICS_RANGE_SECONDS=300
CORE_METRICS_INTERVAL_SECONDS=2

enable (bool)

Enable collecting metrics data of the datarhei Core itself and the FFmpeg processes. The metrics can be queried via the metrics API endpoint.

By default collecting the metrics is disabled.

enable_prometheus (bool)

Enable prometheus endpoint at /metrics. This requires that collecting metrics is enabled.

By default this is disabled.

range_sec (integer)

Define for how many seconds historic metrics data should be kept.

By default this value is set to 300.

interval_sec (integer)

Define in which interval (in seconds) the metrics should be collected.

By default this value is set to 2.

The documentation of the API is available on /api/swagger/index.html

Example:

docker run -d --name core -p 8080:8080 datarhei/core:latest

make init swagger
make run

Public demo:

Golang

Example

Repository

Python3

Install

Example

Repository

The settings for the in-memory filesystem. This filesystem is accessible on /memfs via HTTP. This filesystem can only be accessed via HTTP. Writing to and deleting from the filesystem can be restricted by HTTP basic auth.

Configuration

auth.enable (bool)

Set this value to true in order to enable basic auth for PUT, POST, and DELETE operations on /memfs. Read access (GET, HEAD) is not restricted. If enabled, you have to define a username and a password.

By default this value is set to false.

auth.username (string)

Username for Basic-Auth of /memfs. This has to be set if basic auth is enabled.

By default this value is not set, i.e. an empty string.

auth.password (string)

Password for Basic-Auth of /memfs. This has to be set if basic auth is enabled.

By default this value is not set, i.e. an empty string.

max_size_mbytes (unsigned integer)

The maximum amount of data that is allowed to be stored in this filesystem. The value is interpreted as megabytes. A 507 Insufficient Storage will be returned if you hit the limit. Use a value equal to or smaller than 0 to not set any limits. The limit will be the available memory.

By default no limit is set, i.e. a value of 0.

purge (bool)

Whether to automatically remove the oldest files if the filesystem is full.

By default this value is set to false.

Logging settings for the datarhei Core.

Configuration

level (string)

The verbosity of the logging. The datarhei Core is writing the logs to stderr. Possible values are:

• silent No logging at all.

• error Only errors will be logged.

• warn Warnings and errors will be logged.

• info General information, warnings, and errors will be logged.

• debug Debug messages and every thing else will be logged. This is very chatty.

The default logging level is info.

topics (array)

Logging topics allow you to restrict what type of messages will be logged. This is practical if you enable debug logging and want to see only the logs you're interested in. An empty list of topics means that all topics will be logged.

An non-exhaustive list of logging topics:

• cleanup

• config

• core

• diskfs

• http

• httpcache

• https

• let's encrypt

• memfs

• process

• processstore

• rtmp

• rtmp/s

• rtmps

• update

• service

• session

• sessionstore

• srt

By default all topics are logged.

max_lines (integer)

The log is also kept in memory for retrieval via the API. This value defines how many lines shoul dbe kept in memory.

The default is 1000 lines.

Restreamer-UI

The Restreamer-UI is an easy interface to manage multiple live and restreams on different platforms like the website, YouTube, Facebook and more.

Quick start

1. Start the Core

2. Start the Interface

3. Open the Interface and connect it to the running Core

Requirements

• Go v1.18+ ()

Build

Clone the repository and build the binary:

After the build process, the binary is available as core

For more build options, run make help.

Repository

Cross Compile

If you want to run the binary on a different operating system and/or architecture, you create the appropriate binary by simply setting some environment variables, e.g.

Docker

Build the Docker image and run it to try out the API with FFmpeg

How to customize FFmpeg

Code style

The source code is formatted with go fmt, or run make fmt. Static analysis of the source code is done with staticcheck (see ), or run make lint.

Before committing changes, you should run make commit to ensure that the source code is in shape.

The datarhei Core includes a simple SRT server for publishing and playing streams. Check out the and the . This API endpoint will list the details of all currently publishing and playing streams.

This class of endpoints provide access to information that can be used for widgets in a website. These endpoints are not protected by the API access control.

Process

Fetch minimal statistics about a process. You need to know the process ID.

Settings for static HTTP routes.

Configuration

blocked_prefixes (array)

List of path prefixes that are not allowed to be overwritten by a static route. If a static route would overwrite one of the blocked prefixes, an error will be thrown at startup. For the environment variable, provide a comma-separated list of prefixes, e.g. /prefix1,/prefix2.

By default this value is set to ["/api"].

routes (map)

A list of static routes. This maps a path to a different path and results in a HTTP redirect, e.g. {"/foo.txt": "/bar.txt"} will redirect requests from /foo.txt to /bar.txt. Path have to start with a / and they are based on on the filesystem.

The special suffix /* of a route allows you to serve whole directories from another root than storage.disk.dir, e.g. {"/ui/*", "/path/to/ui"}. If you use a relative path as target, then it will be added to the current working directory.

By default no routes are defined.

ui_path (string)

A path to a directory holding UI files. This will be mounted as /ui.

By default this value is not set, i.e. an empty string.

The settings for the built-in SRT server. Check out our RTMP guide for more information.

Configuration

{
   "rtmp": {
      "enable": false,
      "enable_tls": false,
      "address": ":1935",
      "address_tls": ":1936",
      "app": "/",
      "token": ""
   }
}

CORE_RTMP_ENABLE=false
CORE_RTMP_ENABLE_TLS=false
CORE_RTMP_ADDRESS=":1935"
CORE_RTMP_ADDRESS_TLS=":1936"
CORE_RTMP_APP="/"
CORE_RTMP_TOKEN=

enable (bool)

Set this value to true in order to enable the built-in RTMP server.

By default the RTMP server is disabled.

enable_tls (bool)

Set this value to true to enable the RTMPS server that will run in parallel with the RTMP server on a different port. You have to have tls.enable set to true in order for enabling the RTMPS server because it will use the same certificate as for the HTTPS server.

By default TLS is disabled.

address (string)

If the RTMP server is enabled, it will listen on this address. The default address is :1935.

address_tls (string)

If the RTMPS server is enabled, it will listen on this address. The default address is :1936.

app (string)

Define the app a stream can be published on, e.g. /live to require the path in an RTMP URLs to start with /live.

The default app is /.

token (string)

To prevent anybody from publish or playing streams, define token to be a secret only known to the publishers and subscribers. The token has to be put in the query of the stream URL, e.g. /live/stream?token=abc123.

As of version 16.12.0 the token can be appended to the path instead of a query parameter, e.g. /live/stream/abc123. With this the token corresponds to a stream key.

By default the token is not set (i.e. an empty string).

Settings for session capturing. Sessions for HLS, RTMP, SRT, HTTP, and FFmpeg are captured.

Configurations

{
   "sessions": {
      "enable": true,
      "ip_ignorelist": [
         "127.0.0.1/32",
         "::1/128"
      ],
      "session_timeout_sec": 30,
      "persist": false,
      "persist_interval_sec": 300,
      "max_bitrate_mbit": 0,
      "max_sessions": 0
   }
}

CORE_SESSIONS_ENABLE=false
CORE_SESSIONS_IP_IGNORELIST="127.0.0.1/32,::1/128"
CORE_SESSIONS_SESSION_TIMEOUT_SEC=30
CORE_SESSIONS_PERSIST=false
CORE_SESSIONS_PERSIST_INTERVAL_SEC=300
CORE_SESSIONS_MAXBITRATE_MBIT=0
CORE_SESSIONS_MAXSESSIONS=0

enable (bool)

Set this value to true in order to enable session capturing.

By default this value is set to true.

ip_ignorelist (array)

List of IP ranges in CIDR notation to ignore for session capturing. If either end of a connection falls into this list of IPs, the session will not be captured. For the environment variable provide a comma-separated list of IP ranges in CIDR notation.

By default this value is set to ["127.0.0.1/32","::1/128"].

session_timeout_sec (integer)

The timeout in seconds for an idle session. After this timeout the session is considered as closed. Applies only to HTTP and HLS sessions.

By default this value is set to 30 seconds.

persist (bool)

Whether to persist the session history. The session history is stored as sessions.json in db.dir. If the session history is not persisted it will be kept only in memory.

By default the session history is not persisted, i.e. this value is set to false.

persist_interval_sec (integer)

Interval in seconds in which to persist the current session history. This setting has only effect if persisting the session history is enabled.

By default this value is set to 300 seconds.

max_bitrate_mbit (unsigned integer)

The maximum allowed outgoing bitrate in mbit/s. If the limit is reached, any new HLS sessions will be rejected. A value of 0 means no limitation of the outgoing bitrate.

By default this is value is set to 0, i.e. unlimited.

max_sessions (unsigned integer)

The maximum allowed number of simultaneous sessions. If the limit is reached, any new HLS sessions will be rejected. A value of 0 means no limitation of the number of sessions.

By default this value is set to 0, i.e. unlimited.

The datarhei Core includes a simple SRT server for publishing and playing streams. It is not enabled by default. You have to enable it in the config in the srt section or via the corresponding environment variables.

SRT is a modern live streaming protocol with a low latency and network failure tolerance. Read more.

The SRT server supports publishing and requesting streams, similar to an RTMP server. With your SRT client you have to connect to the SRT server always in caller mode and live transmission mode.

Example:

srt://127.0.0.1:6000?mode=caller&transmode=live&streamid=...

Passphrase

If a passphrase is set in the config (or via environment variable), you have to provide the passphrase in the SRT URL. Example SRT URL with the passphrase foobarfoobar:

srt://127.0.0.1:6000?mode=caller&transmode=live&streamid=...&passphrase=foobarfoobar

StreamID

In order to define whether you want to publish or request a resource, you have to provide your intent in the streamid.

The streamid is formatted as follows:

[resource],mode:[request|publish],token:[token]

The resource is the name of the stream. This can be anything. You can publish only one stream with that same name.

The mode is either request or publish. If you don't provide a mode, request will be assumed. You can only request resources that are currently publishing. You can only publish resources that are not already publishing.

The token is the one defined in the config (see srt.token). If no token is configured, you can omit the token in the streamid.

Examples

Publishing the resource 12345 with the token foobar:

12345,mode:publish,token:foobar

Publishing the resource 12345 with no token defined in the configuration:

12345,mode:publish

Requesting the resource 12345 with no token defined in the configuration:

12345

Requesting the resource 12345 with the token foobar:

12345,token:foobar

The whole SRT URL might look like this for the last example:

srt://127.0.0.1:6000?mode=caller&transmode=live&streamid=12345,token:foobar

API

Via the SRT endpoint in the API you can gather statistics about the currently connected SRT clients.

Settings for the host datarhei Core is running on.

Configuration

{
   "host": {
        "name": ["domain.com"],
        "auto": true
   }
}

CORE_HOST_NAME=domain.com
CORE_HOST_AUTO=true

name (array)

A list of public host/domain names or IPs this host is reacheable under. For the ENV use a comma separated list of public host/domain names or IPs.

The default is an empty list.

auto (bool)

Enable detection of public IP addresses in case the list of names is empty.

By default this is set to true.

Sessions are tracking user actions regarding pushing and pulling video data to and from the core. There are different kind of sessions that are captured:

• HLS Sessions (hls collector) How many users are watching an HLS stream

• HLS Ingress Sessions (hlsingress collector) How many users are publishing a stream via the in-memory filesystem

• HTTP Sessions (http collector) How many users are reading or writing http data via the build-in HTTP server

• RTMP Sessions (rtmp collector) How many users are publishing or watching a stream via the built-in RTMP server

• SRT Sessions (srt collector) How many users are publishing or watching a stream via the built-in SRT server

• FFmpeg Sessions (ffmpeg collector) How many streams is FFmpeg using as an input or as an output

The data for each session include the current ingress and egress bandwidth, the total amount of data, an identifier for the session itself, the local end of the stream and the remote end of the stream.

The following API endpoints allow to extract information about the currently active sessions or additionally a summary of the already finished sessions. Each endpoint requires a comma-separated list of collectors as query parameter collectors.

Active sessions with summary

Example:

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_sessions = client.v3_session_get(
    collectors="hls,rtmp"
)
print(core_sessions)

Description:

Active sessions

Example:

curl http://127.0.0.1:8080/api/v3/session/active?collectors=hls,rtmp \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -X GET

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_active_sessions = client.v3_session_get_active(
    collectors="hls,rtmp"
)
print(core_active_sessions)

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

sessions, err := client.SessionsActive([]string{"hls", "rtmp"})
fmt.Printf("%+v\n", sessions)

Description:

Raspberry Pi 4

Date: 12.02.2021 Version: 2.1.0 (non-public release)

Goal: Bandwidth Limitation Test (HLS Sessions)

Settings

CORE_SESSION_BANDWIDTH_LIMIT=800

sysctl -w net.core.netdev_budget_usecs=60000
sysctl -w net.core.netdev_budget=900
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=2621440

echo "6" > /proc/irq/33/smp_affinity
echo "1-2" > /proc/irq/33/smp_affinity_list
echo c > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo c > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo c > /sys/class/net/eth0/queues/tx-0/xps_cpus
echo c > /sys/class/net/eth0/queues/tx-1/xps_cpus
echo c > /sys/class/net/eth0/queues/tx-2/xps_cpus
echo c > /sys/class/net/eth0/queues/tx-3/xps_cpus
echo c > /sys/class/net/eth0/queues/tx-4/xps_cpus
/sbin/ethtool -K eth0 rx-checksum on
/sbin/ethtool -K eth1 rx-checksum on

Results

The datarhei Core includes a simple RTMP server for publishing and playing streams. Check out the RTMP configuration and the RTMP guide. This API endpoint will list the names of all currently publishing streams.

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_rtmp = client.v3_rtmp_get()
print(core_rtmp)

Settings for accessing the available storage types. The storages are accessible via HTTP, mounted to different paths.

Configuration

{
   "storage": {
     "mimetypes_file": "./mime.types",
     "cors": {
         "origins": [
            "*"
         ]
     },
     "disk": {...},
     "memory": {...},
     "s3": []
   }
}

CORE_STORAGE_MIMETYPES_FILE="./mime.types"
CORE_STORAGE_CORS_ORIGINS="*"

mimetypes_file (string)

Path to a file with the mime-type definitions. This is a file with the MIME types has one MIME type per line followed by a list of file extensions (including the "."). Files served from the storages will have the matching mime-type associated to it.

Example:

text/plain  .txt
text/html   .htm .html
...

Default: ./mime.types

cors.origins (array)

Define a list of allowed CORS origin for accessing the storages.

By default it contains the only element *, allowing access from anywhere.

Disk

The disk storage is mounted at / via the HTTP server.

In-memory

The memory storage is mounted at /memfs via the HTTP server.

S3

The S3 storage is mounted at the configured path via the HTTP server.

The API allows you to manipulate the contents of the available filesystems.

The settings for the S3 filesystem. This filesystem is accessible on the configured path via HTTP. This filesystem can only be accessed via HTTP. Writing to and deleting from the filesystem can be restricted by HTTP basic auth.

Any S3 compatible service can be used, e.g. Amazon, Minio, Backblaze, ...

Configuration

name (string)

The name for this storage. The name will be used in placeholders, e.g. {fs:aws}, and for accessing the filesystem via the api, e.g. /api/v3/fs/aws. The name memfs is reserved for the in-memory filesystem.

By default this value is not set, but is required.

mountpoint (string)

The path where the filesystem will be mounted in the HTTP server. It needs to be an absolute path. A mountpoint is required.

By default this value is not set, but is required.

auth.enable (bool)

Set this value to true in order to enable basic auth for PUT, POST, and DELETE operations on the configured mountpoint. Read access (GET, HEAD) is not restricted. If enabled, you have to define a username and a password.

By default this value is set to false.

auth.username (string)

Username for Basic-Auth of the configured mountpoint. This has to be set if basic auth is enabled.

By default this value is not set, i.e. an empty string.

auth.password (string)

Password for Basic-Auth of the configured mountpoint. This has to be set if basic auth is enabled.

By default this value is not set, i.e. an empty string.

endpoint (string)

The endpoint for the S3 storage. For Amazon AWS S3 it would be e.g. s3.amazonaws.com. Ask your S3 storage provider to the necessary credentials.

By default this value is not set, i.e. an empty string.

access_key_id (string)

Your access key ID for the S3 storage. Ask your S3 storage provider to the necessary credentials.

By default this value is not set, i.e. an empty string.

secret_access_key (string)

Your secret access key for the S3 storage. Ask your S3 storage provider to the necessary credentials.

By default this value is not set, i.e. an empty string.

bucket (string)

The name of the bucket you want to use. If the bucket does not exist already it will be created.

By default this value is not set, i.e. an empty string.

region (string)

Identifier for the region the storage is located, e.g. eu, us-west1, ... . If your S3 storage provider doesn't support regions, leave this field empty.

By default this value is not set, i.e. an empty string.

use_ssl (bool)

Whether to use HTTPS or HTTP. It is strongly recommended to enable this setting.

By default this is set to false.

The settings for the built-in SRT server. Check out our for more information.

Configuration

enable (bool)

Set this value to true in order to enable the built-in SRT server.

By default the SRT server is disabled.

address (string)

If the SRT server is enabled, it will listen on this address. The default address is :6000.

passphrase (string)

Define a passphrase in order to enable SRT encryption. If the passphrase is set it is required and applies to all connections.

By default the passphrase is not set (i.e. an empty string).

token (string)

The token is an arbitrary string that needs to be comunicated in the streamid. Only with a valid token it is possible to publish or request streams. If the token is not set, anybody could publish and request streams.

By default the token is not set (i.e. an empty string).

log.enable (bool)

Set this value to true in order to enable logging for the SRT server. This will log events on the SRT protocol level. You have to provide the topics you are interested in, otherwise nothing will be logged.

By default the logging is disabled.

log.topic (array)

Logging topics allow you to define what type of messages will be logged. This is practical if you want to debug a SRT connection. An empty list of topics means that no topics will be logged.

Find a list of known logging topics on the .

By default no topics are logged (i.e. an empty array).

The last log events are kept in memory and are accessible via the /api/v3/log endpoint. You can either retrieve the log lines in the format the Core is writing them to the console (?format=console) or in raw format (?format=raw). By default they are returned in "console" format.

You can define the number of last log lines kept in memory by either setting an appropriate value in the config (log.max_lines) or via an enviroment variable (CORE_LOG_MAX_LINES).

Read

Example:

Description:

These are the settings for securing the API from unwanted access.

Configuration

{
   "api": {
      "read_only": false,
      "access": {
         "http": {
            "allow": [],
            "block": []
         },
         "https": {
            "allow": [],
            "block": []
         }
      },
      "auth": {
         "enable": fals,
         "disable_localhost": false,
         "username": "",
         "password": "",
         "jwt": {
            "secret": ""
         },
         "auth0": {
            "enable": false,
            "tenants": []
         }
      }
   }
}

CORE_API_READ_ONLY=false
CORE_API_ACCESS_HTTP_ALLOW="127.0.0.1/32,::1/128"
CORE_API_ACCESS_HTTP_BLOCK="127.0.0.1/32,::1/128"
CORE_API_ACCESS_HTTPS_ALLOW="127.0.0.1/32,::1/128"
CORE_API_ACCESS_HTTPS_DENY="127.0.0.1/32,::1/128"
CORE_API_AUTH_ENABLE=true
CORE_API_AUTH_DISABLE_LOCALHOST=true
CORE_API_AUTH_USERNAME=
CORE_API_AUTH_PASSWORD=
CORE_API_AUTH_JWT_SECRET=mySecret
CORE_API_AUTH_AUTH0_ENABLE=false
CORE_API_AUTH_AUTH0_TENANTS=

read_only (bool)

Set this value to true in order to allow only ready access to the API. All API endpoints for writing will not be mounted.

By default this value is set to false.

access.http.allow (array)

A list of IPs that are allowed to access the API via HTTP. Each entry has to be an IP range in CIDR notation, e.g. ["127.0.0.1/32","::1/128"]. Provide the list as comma-separated values for the environment variable, e.g. 127.0.0.1/32,::1/128. If the list is empty, then all IPs are allowed. If the list contains any invalid IP range, the server will refuse to start.

By default the list is empty.

access.http.block (array)

A list of IPs that are not allowed to access the API via HTTP. Each entry has to be an IP range in CIDR notation. Provide the list as comma-separated values for the environment variable. If the list is empty, then no IPs will be blocked. If the list contains any invalid IP range, the server will refuse to start.

By default the list is empty.

access.https.allow (array)

A list of IPs that are allowed to access the API via HTTPS. Each entry has to be an IP range in CIDR notation. Provide the list as comma-separated values for the environment variable. If the list is empty, then all IPs are allowed. If the list contains any invalid IP range, the server will refuse to start.

By default the list is empty.

access.https.block (array)

A list of IPs that are not allowed to access the API via HTTPS. Each entry has to be an IP range in CIDR notation. Provide the list as comma-separated values for the environment variable. If the list is empty, then no IPs will be blocked. If the list contains any invalid IP range, the server will refuse to start.

By default the list is empty.

auth.enable (bool)

Set this value to true to enable JWT authentication for the API. If it is enabled, you have to provide a username and password. The username and password will be sent to the /api/login endpoint in order to obtain an access and refresh JWT.

By default this value is set to false.

auth.disable_localhost (bool)

Set this value to true in order to allow unprotected access from localhost.

Be default this value is set to false.

auth.username (string)

The username for JWT authentication. If JWT authentication is enabled, a username must be defined.

By default this value is empty, i.e. no username defined.

auth.password (string)

The password for JWT authentication. If JWT authentication is enabled, a password must be defined.

By default this value is empty, i.e. no password defined.

auth.jwt.secret (string)

A secret for signing the JWT. If you leave this value empty, a random secret will be generated for you.

By default this value is empty.

auth.auth0.enable (bool)

Set this value to true in order to enable API auth0 protection. With this a valid Auth0 access JWT can be used instead of a username/password in order to obtain the access and refresh JWT. Additionally you have to provide a list of tenants and their users to validate the Auth0 access JWT against.

By default this value is set to false.

auth.auth0.tenants (array)

A list of allowed tenants and their users. A tenant is a JSON object:

{
    "domain": "",
    "audience": "",
    "clientid": "",
    "users": [],
}

You can obtain the domain, audience, and clientid from your Auth0 account. You also have to provide a list of allowed users that are member of that tenant.

For providing the list of tenants and their users as an environement variable you have to provide a comma-separated list of base64 encoded tenant JSON objects.

As of version 16.12.0 there's a different syntax available for providing the tenants as environment variable. A list of comma separated URLs of this form:

auth0://[clientid]@[domain]?aud=[audience]&user=...&user=...

By default this list is empty.

The Core-FFmpeg bundle uses Docker's multi-stage process so that FFmpeg and the Core can be updated and maintained independently.

When building the Core-FFmpeg bundle, an FFmpeg image is used. The previously created Golang libraries and folder structures are copied into this image.

This process speeds up the creation of the final Core-FFmpeg bundle, as existing or previously created images can be used, and compiling all the code is no longer required.

The following base images are available:

• docker.io/datarhei/base:alpine-ffmpeg-latest

• docker.io/datarhei/base:alpine-ffmpeg-rpi-latest

• docker.io/datarhei/base:ubuntu-ffmpeg-cuda-latest

• docker.io/datarhei/base:ubuntu-ffmpeg-vaapi-latest

• docker.io/datarhei/base:alpine-core-latest

• docker.io/datarhei/base:ubuntu-core-latest

Specific versions are available on the Docker website:

1. Create a custom FFmpeg image

1.1 Clone the FFmpeg build files

git clone github.com/datarhei/ffmpeg

Repository

1.2 Switch to the cloned folder

cd ffmpeg

1.2 Change a Dockerfile

...
RUN cd /dist/ffmpeg-${FFMPEG_VERSION} && \
  patch -p1 < /contrib/ffmpeg-jsonstats.patch && \
  patch -p1 < /contrib/ffmpeg-hlsbitrate.patch && \
  ./configure \
  --extra-version=datarhei \
  --prefix="${SRC}" \
  --extra-libs="-lpthread -lxml2 -lm -lz -lsupc++ -lstdc++ -lssl -lcrypto -lz -lc -ldl" \
  --enable-nonfree \
  --enable-gpl \
  --enable-version3 \
  --enable-postproc \
  --enable-static \
  --enable-openssl \
  --enable-libxml2 \
  --enable-libv4l2 \
  --enable-v4l2_m2m \
  --enable-libfreetype \
  --enable-libsrt \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvpx \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --disable-ffplay \
  --disable-shared && \
  make -j$(nproc) && \
  make install
...

1.3 Build a custom image

./Build.sh default

Arguments:

• default Dockerfile: Dockerfile.alpine Image name: datarhei/base:alpine-ffmpeg-latest

• rpi Dockerfile: Dockerfile.alpine.rpi Image name: datarhei/base:alpine-ffmpeg-rpi-latest

• cuda Dockerfile: Dockerfile.ubuntu.cuda Image name: datarhei/base:ubuntu-ffmpeg-cuda-latest

• vaapi Dockerfile: Dockerfile.ubuntu.vaapi Image name: datarhei/base:alpine-ffmpeg-vaapi-latest

2. Create a custom Core image

2.1 Clone the Core build files

git clone [email protected]:datarhei/core.git

Repository

2.2 Switch into the cloned folder

cd core

2.3 Build a custom image

docker build -t datarhei/base:alpine-core-latest .

3. Create a custom Core-FFmpeg bundle

You can find the Dockerfile for the bundle (Dockerfile.bundle) in the cloned Core repository.

3.1 Build a custom image

docker build \
    -f Dockerfile.bundle \
    --build-arg CORE_IMAGE=datarhei/base:alpine-core-latest \
    --build-arg FFMPEG_IMAGE=datarhei/base:alpine-ffmpeg-latest \
    -t core-bundle:dev .

Skills denote the capabilities of the used FFmpeg binary. It includes version information, supported input and output protocols, available hardware accelerators, supported formats for muxing and demuxing, filters, and available input and output devices.

Read

Example:

curl http://127.0.0.1:8080/api/v3/skills \
   -H 'accept: application/json' \
   -X GET

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_ffmpeg_skills = client.v3_skills_get()
print(core_ffmpeg_skills)

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

skills, err := client.Skills()
fmt.Printf("%+v\n", skills)

Description:

Reload

Reloading the skills might be necessary if you plug e.g. an USB device. It will only show up in the list of available devices if they are probed again.

Example:

curl http://127.0.0.1:8080/api/v3/skills/reload \
   -H 'accept: application/json' \
   -X GET

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

client.v3_skills_reload()

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

err := client.SkillsReload()
if err != nil {
    ...
}

skills, err := client.Skills()
fmt.Printf("%+v\n", skills)

Helping Hands

To ensure your project's success, we offer installation services and ongoing support from the datarhei team. Helping Hands includes installing a datarhei Restreamer or datarhei Core on a server instance and managing the server as a managed service.

1. Installation

2. Configuration

3. Updates

4. Fix errors

Communication and assistance can be accessed through email or chat on our Discord and conducted in English and German. We appreciate your support of "Helping Hands."

Within 48 hours of becoming a patron or sponsor on Open Collective, a representative from FOSS GmbH will contact you for further information regarding the installation process.

Service level agreements are in effect for the duration of the active donation.

If you have questions about the Helping Hands Service Level Agreement (SLA), contact us on Discord or by email at [email protected]. We're here to help!"

Book now

Business Inquiries

We're the team at FOSS GmbH (https://foss-gmbh.ch) from Switzerland, the creators of datarhei.com. We'll do our best to get back to you within 1-2 days during our regular business hours, excluding weekends and holidays in Switzerland and Germany. No matter what you need help with, we're here to provide professional support, software development, and consulting for anything related to datarhei software.

If you have a commercial request, such as a bug fix or feature enhancement, just let us know, and we'll be happy to provide a quote.

Business contact

Please fill out this form to help us respond to your request as quickly as possible. We're able to assist you in both German and English. However, please remember that we cannot process any requests not submitted through this form.

Community

Private and non-commercial requests can be discussed and resolved on GitHub and Discord's public channels.

Open issue on GitHub

Say hello on Discord

Donation

Every little bit helps! Consider becoming a patron on Patreon or backer on Open Collective to support keeping the software free.

Rate on Google

💙💚🧡💜 Help datarhei with a rating or review on Google. #feelsgoodtobeloved

>> https://t.co/ZT9pRX1gG7

💛 Thanks for using datarhei software. Your datarhei.com team

The settings for the disk filesystem. This filesystem is accessible on / via HTTP. This filesystem can only be accessed for reading via HTTP. Writing to and deleting from the filesystem is possible via the API.

Configuration

{
   "storage": {
      "disk": {
         "dir": "./data",
         "max_size_mbytes": 0,
         "cache": {
            "enable": true,
            "max_size_mbytes": 0,
            "ttl_seconds": 300,
            "max_file_size_mbytes": 1,
            "types": {
               "allow": [],
               "block": []
            }
         }
      }
   }
}

CORE_STORAGE_DISK_DIR="./data"
CORE_STORAGE_DISK_MAXSIZEMBYTES=0
CORE_STORAGE_DISK_CACHE_ENABLE=true
CORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES=0
CORE_STORAGE_DISK_CACHE_TTLSECONDS=300
CORE_STORAGE_DISK_CACHE_MAXFILESIZEMBYTES=1
CORE_STORAGE_DISK_CACHE_TYPES_ALLOW=
CORE_STORAGE_DISK_CACHE_TYPES_BLOCK=".m3u8 .mpd"

dir (string)

Path to a directory on disk. It will be exposed on / for reading.

By default it is set to ./data.

max_size_mbytes (unsigned integer)

The maximum amount of data that is allowed to be stored in this filesystem. The value is interpreted as megabytes. A 507 Insufficient Storage will be returned if you hit the limit. Use a value equal to or smaller than 0 to not impose any limits. Then all the available space on the disk is the limit.

By default no limit is set, i.e. a value of 0.

cache.enable (bool)

Set this value to true in order to enable the cache for the disk. The cache is an LRU cache, i.e. the least recently accessed files in the cache will be removed in case the cache is full and a new file wants to be added.

By default the value is set to true.

cache.max_size_mbytes (unsigned integer)

Limit for the size of the cache in megabytes. A value of 0 doesn't impose any limit. The limit will be the available memory.

By default no limit is set, i.e. a value of 0.

cache.ttl_seconds (integer)

Number of seconds to keep a file in the cache.

By default this value is set to 300 seconds.

cache.max_file_size_mbytes (unsigned integer)

Limit for the size of a file to be allowed to be put in the cache in megabytes.

By default this value is set to 1 megabyte.

cache.types.allow (array)

A list of file extensions to cache, e.g. [".ts", ".mp4"]. Leave the list empty in order to cache all files. Use a space-separated list of extensions for the environment variable, e.g. .ts .mp4.

By default the list is empty.

cache.types.block (array)

A list of file extensions not to cache, e.g. [".m3u8", ".mpd"]. Leave the list empty in order to block no extension. Use a space-separated list of extensions for the environment variable, e.g. .m3u8 .mpd.

By default the manifest files for HLS and DASH are blocked from caching, i.e. [".m3u8", ".mpd"].

Probing a process means to detect the vitals of the inputs of a process, e.g. frames, bitrate, codec, ... for each input and stream, e.g. a video file on disk may contain two video streams (low and high resolution, audio and subtitle streams in different languages).

A process must already exists before it can be probed. During probing only the global FFmpeg options and the inputs are used to construct the FFmpeg command line.

The probe returns an object with an array of the detected streams and any array of lines from the output from the ffmpeg command.

In the following example we assume the process config with these inputs for the process test. Parts that are not relevant for probing have been left out for brevity:

{
    "options": ["-err_detect", "ignore_err", "-y"],
    "input": [
      {
        "address": "testsrc2=rate=25:size=640x360",
        "id": "input_0",
        "options": ["-f", "lavfi", "-re"]
      },
      {
        "address": "anullsrc=r=44100:cl=stereo",
        "id": "input_1",
        "options": ["-f", "lavfi"]
      }
    ]
}

The expected result would be:

{
  "log": [
    "ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers",
    "  built with Apple clang version 14.0.0 (clang-1400.0.29.102)",
    "  configuration: --prefix=/usr/local/Cellar/ffmpeg/5.1.2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox",
    "  libavutil      57. 28.100 / 57. 28.100",
    "  libavcodec     59. 37.100 / 59. 37.100",
    "  libavformat    59. 27.100 / 59. 27.100",
    "  libavdevice    59.  7.100 / 59.  7.100",
    "  libavfilter     8. 44.100 /  8. 44.100",
    "  libswscale      6.  7.100 /  6.  7.100",
    "  libswresample   4.  7.100 /  4.  7.100",
    "  libpostproc    56.  6.100 / 56.  6.100",
    "Input #0, lavfi, from 'testsrc2=rate=25:size=640x360':",
    "  Duration: N/A, start: 0.000000, bitrate: N/A",
    "  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn",
    "Input #1, lavfi, from 'anullsrc=r=44100:cl=stereo':",
    "  Duration: N/A, start: 0.000000, bitrate: 705 kb/s",
    "  Stream #1:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s",
    "At least one output file must be specified"
  ],
  "streams": [
    {
      "bitrate_kbps": 0,
      "channels": 0,
      "codec": "rawvideo",
      "coder": "",
      "duration_sec": 0,
      "format": "lavfi",
      "fps": 0,
      "height": 360,
      "index": 0,
      "language": "und",
      "layout": "",
      "pix_fmt": "yuv420p",
      "sampling_hz": 0,
      "stream": 0,
      "type": "video",
      "url": "testsrc2=rate=25:size=640x360",
      "width": 640
    },
    {
      "bitrate_kbps": 705,
      "channels": 0,
      "codec": "pcm_u8",
      "coder": "",
      "duration_sec": 0,
      "format": "lavfi",
      "fps": 0,
      "height": 0,
      "index": 1,
      "language": "und",
      "layout": "stereo",
      "pix_fmt": "",
      "sampling_hz": 44100,
      "stream": 0,
      "type": "audio",
      "url": "anullsrc=r=44100:cl=stereo",
      "width": 0
    }
  ]
}

The field index refers to the input and the field stream refers to the stream of an input.

Example: probe the inputs of a process with the ID test:

curl http://127.0.0.1:8080/api/v3/process/test/probe \
   -H 'accept: application/json' \
   -X GET

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080"
)
client.login()

core_process_test_probe = client.v3_process_get_probe(
    id="test"
)
print(core_process_test_probe)

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

probe, err := client.ProcessProbe("test")
if err != nil {
    ...
}

fmt.Printf("%+v\n", probe)

Description:

The disk filesystem gives access to the actual directory that has been provided in the configuration as . It accessible only for retrieval via HTTP under /.

Given that the requested file exists, the returned Content-Type is based solely on the file extension. For a list of known mime-types and their extensions see in the configuration.

Access

API

The contents of the disk filesystem at / are also accessible via the API in the same way as described above, but with the same protection as the API (see ) for all operations. It is also possible to list all files that are currently in the filesystem.

Create, Update

Example:

After the successful upload the file is available at /example.txt and /api/v3/fs/disk/example.txt.

Description:

Read

List all files

Listing all currently stored files is done by calling /api/v3/fs/disk. It also accepts the query parameters pattern, sort (name, size, or lastmod) and order (asc or desc). If none of the parameters are given, all files will be listed sorted by their last modification time in ascending order.

With the pattern parameter you can filter the list based on a , with the addition of the ** placeholder to include multiple subdirectories, e.g. listing all .ts file in the root directory has the pattern /*.ts, listing all .ts file in the whole filesystem has the pattern /**.ts.Example:

Description:

Download a file

For downloading a file you have to specify the complete path and filename. The Content-Type will always be application/data.

Example:

Description:

Delete

For deleting a file you have to specify the complete path and filename.

Example:

Description:

Configuration

A/V Processing

1. Core launches and monitors FFmpeg processes

2. FFmpeg can use HTTP, RTMP, and SRT services as streaming backends for processing incoming and outgoing A/V content.

3. Several storage locations are available for the HTTP service: In-memory file system, aka MemFS (very fast without disk I/O.) Disk file system, aka DiskFS, for storage on the HDD/SSD of the host system.

4. Optionally, FFmpeg can access host system devices such as GPU and USB interfaces (requires FFmpeg built-in support).

These settings are for configuring the TLS / HTTPS support for datarhei Core.

Configuration

{
   "tls": {
      "address": ":8181",
      "enable": false,
      "auto": false,
      "mail": "[email protected]",
      "cert_file": "",
      "key_file": "",
   }
}

CORE_TLS_ADDRESS=":8181"
CORE_TLS_ENABLE=false
CORE_TLS_AUTO=false
[email protected]
CORE_TLS_CERT_FILE=
CORE_TLS_KEY_FILE=

address (string)

If TLS is enabled, the HTTPS server will listen on this address. The default address is :8181.

enable (bool)

Set this value to true in order to enable TLS / HTTPS support. If enabled you have to either provide your own certificate (see cert_file and key_file) or enable automatic certificate from Let's Encrypt (see auto).

If TLS is enabled, a HTTP server listening on address will be additionally started. This server provides access to everything as the HTTPS server, additionally it will allow ACME http-1 challenges in case Let's Encrypt (auto) certificates are enabled.

By default this is set to false.

auto (bool)

Enable automatic certificate generation from Let's Encrypt. This only works if enable is set to true and at least one public hostname is defined in host.name. All listed hostnames will be included in the certificate. All listed public hostnames is required to point to the host where datarhei Core is running on.

The obtained certificates will be stored in the /cert subdirectory of db.dir to be available after a restart.

Any provided paths in cert_file and key_file will be ignored.

By default this is set to false.

mail (string)

An email address that is required for Let's Encrypt in order to receive a certificate.

By default the email address [email protected] is used.

cert_file (string)

If you bring your own certificate, provide the path to the certificate file in PEM format.

By default this is not set.

key_file (string)

If you bring your own certificate, provide the path to the key file in PEM format

By default this is not set.

Examples

Let's Encrypt

If you want to use automatic certificates from Let's Encrypt, set tls.enable and tls.auto to true. and host.name has to be set to the domain name this host will be reachable. Otherwise the ACME http-1 challenge will not work.

{
   "address": ":80",
   "host": {
      "name": ["domain.com"],
      "auto": false
   },
   "tls": {
      "address": ":8181",
      "enable": true,
      "auto": true,
      "mail": "[email protected]"
   }
}

CORE_ADDRESS=:80
CORE_HOST_NAME=domain.com
CORE_HOST_AUTO=false
CORE_TLS_ADDRESS=:8181
CORE_TLS_ENABLE=true
CORE_TLS_AUTO=true
[email protected]

Self-Signed certificates

To create a self-signed certificate and key file pair, run this command and provide a reasonable value for the Common Name (CN). The CN is the fully qualified name of the host the instance is running on (e.g., localhost). You can also use an IP address or a wildcard name, e.g., *.example.com.

RSA SSL certificate

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out cert.pem -subj '/CN=localhost'

ECDSA SSL certificate

openssl ecparam -name secp521r1 -genkey -out key.pem
openssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj '/CN=localhost'

Call openssl ecparam -list_curves to see all available supported curves listed.

{
   "host": {
      "name": ["domain.com"],
      "auto": false
   },
   "tls": {
      "address": ":8181",
      "enable": true,
      "auto": false,
      "cert_file": "/core/config/example.cert",
      "key_file": "/core/config/example.key"
   }
}

CORE_HOST_NAME=domain.com
CORE_HOST_AUTO=false
CORE_TLS_ADDRESS=:8181
CORE_TLS_ENABLE=true
CORE_TLS_AUTO=false
CORE_TLS_CERT_FILE=/core/config/example.cert
CORE_TLS_KEY_FILE=/core/config/example.key

With auth enabled, you have to retrieve a JWT token before you can access the API calls.

Username/password login

Send the username and password, as defined in api.auth.username and api.auth.password, in the body of the request to the /api/login endpoint in order to obtain valid access and refresh JWT.

Example:

curl http://127.0.0.1:8080/api/login \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -X POST \
   -d '{
         "username": "YOUR_USERNAME",
         "password": "YOUR_PASSWORD"
      }'

{
   "access_token": "eyJz93a...k4laUWw",
   "refresh_token": "eyJz93a...k4laUWx"
}

curl http://127.0.0.1:8080/api/ \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer eyJz93a...k4laUWw' \
   -X GET

curl http://127.0.0.1:8080/api/login/refresh \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer eyJz93a...k4laUWx' \
   -X GET

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080",
    username="YOUR_USERNAME",
    password="YOUR_PASSWORD",
)
client.login()

print(client.token())

import "github.com/datarhei/core-client-go/v16"

client, err := coreclient.New(coreclient.Config{
    Address: "http://127.0.0.1:8080",
    Username: "YOUR_USERNAME",
    Password: "YOUR_PASSWORD",
})
if err != nil {
    ...
}

Description:

Auth0 login

Send a valid Auth0 access JWT in the Authorization header to the /api/login endpoint in order to obtain an access and refresh JWT. The Auth0 tenant and the allowed users must be defined in the configuration.

Example:

curl http://127.0.0.1:8080/api/login \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer eyJz93a...k4laUWw' \
   -X POST

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080",
    auth0_token="eyJz93a...k4laUWw",
)
client.login()

import "github.com/datarhei/core-client-go/v16"

client, err := coreclient.New(coreclient.Config{
    Address: "http://127.0.0.1:8080",
    Auth0Token: "eyJz93a...k4laUWw",
})
if err != nil {
    ...
}

JWT refresh

In order to obtain a new access token, use the refresh_token for a call to /api/login/refresh. Example:

curl http://127.0.0.1:8080/api/login/refresh \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleGkiOjg2NDAwLCJleHAiOjE2NzA1Mjc2MjUsImlhdCI6MTY3MDQ0MTIyNSwiaXNzIjoiZGF0YXJoZWktY29yZSIsImp0aSI6IjczM2Q4Y2UxLTY3YjEtNDM3Yy04YzQ1LTM3Yjg4MmZjMWExZiIsInN1YiI6ImFkbWluIiwidXNlZm9yIjoicmVmcmVzaCJ9.3lqZFJeN7ILfM4DTi0-ZJ7kAzqTMR-yRgRl3o89O-jY' \
   -X GET

{
   "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleGkiOjYwMCwiZXhwIjoxNjcwNDQxODI1LCJpYXQiOjE2NzA0NDEyMjUsImlzcyI6ImRhdGFyaGVpLWNvcmUiLCJqdGkiOiJhZWU4OTZhNS05ZThhLTRlMGQtYjk4Zi01NTA3NTUwNzA2YzUiLCJzdWIiOiJhZG1pbiIsInVzZWZvciI6ImFjY2VzcyJ9.xrnIfNZU9Z0nrUxYddpPQOMO7ypHA2vuqrYuAr95elg"
}

from core_client import Client

client = Client(
    base_url="http://127.0.0.1:8080",
    refresh_token="eyJz93a...k4laUWw",
)
client.token_refresh()

accessToken, refreshToken := client.Tokens()

client, err := coreclient.New(coreclient.Config{
    Address: "http://127.0.0.1:8080",
    Username: "YOUR_USERNAME",
    Password: "YOUR_PASSWORD",
    AccessToken: accessToken,
    RefreshToken: refreshToken,
})

Once the refresh token expires, you have to login again with your username and password, or a valid Auth0 token.

Description:

The process report captures the log output from FFmpeg. The output is split up in a prelude and log output. The prelude is everything before the first progress line is printed. Everything after that is part of the log. The progress lines are not part of the report. Each log line comes with a timestamp of when it has been captured from the FFmpeg output.

The process report helps you to analyze any problems with the FFmpeg command line build from the inputs, outputs, and their options in the process configuration.

If the process is running, the FFmpeg logs will be written to the current report. As soon as the process finishes, the report will be moved to the history. The report history is also part of the response of this API endpoint.

You can define the number of log lines and how many reports should be kept in the history in the config for the datarhei Core in the section.

The following is an example of a report.

Example: read the report of a process with the ID test:

Description:

Settings for the FFmpeg binary.

Configuration

binary (string)

Path to the ffmpeg binary. The system's %PATH will be searched for the ffmpeg binary. You can also provide an absolute or relative path to the binary.

By default this value is set to ffmpeg.

max_processes (integer)

The maximum number of simultaneously running ffmpeg instances. Set this value to 0 in order to not impose any limit.

By default this value is set to 0.

access.*

To control where FFmpeg can read from and where FFmpeg can write from, you can define patterns that matches the input addresses or the output addresses. These patterns are regular expressions that can be provided here. For the respective environment variables the expressions need to be space-separated, e.g. https?:// rtsp:// rtmp://.

Independently of the values of access.output there's a check that verifies that output can only be written to the directory specified in and works as follows: if the address has a protocol specifier other than file:, then no further checks will be applied. If the protocol is file: or no protocol specifier is given, the address is assumed to be a path that is checked to be inside of storage.disk.dir.

It will be rejected if the address is outside the storage.disk.dir directory. Otherwise, the protocol file: will be prepended. If you want to explicitely allow or block access to the filesystem, use file: as pattern in the respective list.

Special cases are the output addresses - (which will be rewritten to pipe:), and /dev/null, which will be allowed even though it's outside of storage.disk.dir.

access.input.allow (array)

List of patterns for allowed inputs.

By default this list is empty, i.e. all inputs are allowed.

access.input.block (array)

List of patterns for disallowed inputs.

By default this list is empty, i.e. no inputs are blocked.

access.output.allow (array)

List of patterns for allowed outputs.

By default this list is empty, i.e. all outputs are allowed.

access.output.block (array)

List of patterns for disallowed outputs.

By default this list is empty, i.e. no outputs are blocked.

log.max_lines (integer)

The number of latest FFmpeg log lines for each process to keep.

By default this value is set to 50 lines.

log.max_history (integer)

The number of historic logs for each process to keep.

By default this value is set to 3.

The metadata for a process allows you to store arbitrary JSON data with that process, e.g. if you have an app that uses the datarhei Core API you can store app-specific information in the metadata.

In order not to conflict with other apps that might write to the metadata as well, you have to provide a key under which your metadata is stored. Think of a namespace or similar.

Create, Update

Add or update the metadata for a process and key.

Example: Write a metadata object into the metadata for the process test and key desc.

Description:

Read

Read out the stored metadata. The key is optional. If the key is not provided, all stored metadata for that process will be in the result.

Example: read the metadata object for process test that is stored under the key desc.

Description:

The process state reflects the current vitals of an process. This includes for example the runtime the process is already in this state, the order (if it should be running or stopped, see Command), the current CPU and memory consumption, the actual FFmpeg command line, and some more.

If the process is running you will recieve progress data additionally to the above mentioned metrics. Progress data includes for each input and output stream the bitrate, framerate, bytes read/written, the speed of the processing, duplicated, dropped frames, and so on.

In the following an example output for a running process:

{
  "command": [
    "-err_detect",
    "ignore_err",
    "-y",
    "-f",
    "lavfi",
    "-re",
    "-i",
    "testsrc2=rate=25:size=640x360",
    "-f",
    "lavfi",
    "-i",
    "anullsrc=r=44100:cl=stereo",
    "-dn",
    "-sn",
    "-map",
    "0:0",
    "-codec:v",
    "libx264",
    "-preset:v",
    "ultrafast",
    "-b:v",
    "1024k",
    "-maxrate:v",
    "1024k",
    "-bufsize:v",
    "1024k",
    "-r",
    "25",
    "-sc_threshold",
    "0",
    "-pix_fmt",
    "yuv420p",
    "-g",
    "50",
    "-keyint_min",
    "50",
    "-fps_mode",
    "cfr",
    "-tune:v",
    "zerolatency",
    "-map",
    "1:0",
    "-filter:a",
    "aresample=osr=44100:ochl=stereo",
    "-codec:a",
    "aac",
    "-b:a",
    "64k",
    "-shortest",
    "-flags",
    "+global_header",
    "-tag:v",
    "7",
    "-tag:a",
    "10",
    "-f",
    "tee",
    "[f=hls:start_number=0:hls_time=2:hls_list_size=6:hls_flags=append_list+delete_segments+program_date_time+independent_segments+temp_file:hls_delete_threshold=4:hls_segment_type=fmp4:hls_fmp4_init_filename=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.mp4:hls_fmp4_init_resend=1:hls_segment_filename=http\\\\://admin\\\\:xxx@localhost\\\\:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0_%04d.mp4:master_pl_name=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.m3u8:master_pl_publish_rate=2:method=PUT]http://admin:xxx@localhost:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0.m3u8|[f=flv]rtmp://localhost:1935/live/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.stream?token=foobar|[f=mpegts]srt://localhost:6000?mode=caller\u0026transtype=live\u0026streamid=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf,mode:publish,token:abc"
  ],
  "cpu_usage": 3.022,
  "exec": "running",
  "last_logline": "[hls @ 0x7fa31f810240] Opening 'http://admin:xxx@localhost:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.mp4' for writing",
  "memory_bytes": 21532672,
  "order": "start",
  "progress": {
    "bitrate_kbit": 0,
    "drop": 0,
    "dup": 0,
    "fps": 24.833,
    "frame": 62221,
    "inputs": [
      {
        "address": "testsrc2=rate=25:size=640x360",
        "avstream": null,
        "bitrate_kbit": 0,
        "codec": "rawvideo",
        "coder": "",
        "format": "lavfi",
        "fps": 0,
        "frame": 0,
        "height": 360,
        "id": "input_0",
        "index": 0,
        "packet": 0,
        "pix_fmt": "yuv420p",
        "pps": 0,
        "size_kb": 0,
        "stream": 0,
        "type": "video",
        "width": 640
      },
      {
        "address": "anullsrc=r=44100:cl=stereo",
        "avstream": null,
        "bitrate_kbit": 0,
        "codec": "pcm_u8",
        "coder": "",
        "format": "lavfi",
        "fps": 0,
        "frame": 0,
        "id": "input_1",
        "index": 1,
        "layout": "stereo",
        "packet": 0,
        "pps": 0,
        "sampling_hz": 44100,
        "size_kb": 0,
        "stream": 0,
        "type": "audio"
      }
    ],
    "outputs": [
      {
        "address": "[f=hls:start_number=0:hls_time=2:hls_list_size=6:hls_flags=append_list+delete_segments+program_date_time+independent_segments+temp_file:hls_delete_threshold=4:hls_segment_type=fmp4:hls_fmp4_init_filename=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.mp4:hls_fmp4_init_resend=1:hls_segment_filename=http\\\\://admin\\\\:xxx@localhost\\\\:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0_%04d.mp4:master_pl_name=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.m3u8:master_pl_publish_rate=2:method=PUT]http://admin:xxx@localhost:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0.m3u8|[f=flv]rtmp://localhost:1935/live/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.stream?token=foobar|[f=mpegts]srt://localhost:6000?mode=caller\u0026transtype=live\u0026streamid=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf,mode:publish,token:abc",
        "avstream": null,
        "bitrate_kbit": 0,
        "codec": "h264",
        "coder": "",
        "format": "tee",
        "fps": 0,
        "frame": 0,
        "height": 360,
        "id": "output_0",
        "index": 0,
        "packet": 0,
        "pix_fmt": "yuv420p",
        "pps": 0,
        "size_kb": 0,
        "stream": 0,
        "type": "video",
        "width": 640
      },
      {
        "address": "[f=hls:start_number=0:hls_time=2:hls_list_size=6:hls_flags=append_list+delete_segments+program_date_time+independent_segments+temp_file:hls_delete_threshold=4:hls_segment_type=fmp4:hls_fmp4_init_filename=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.mp4:hls_fmp4_init_resend=1:hls_segment_filename=http\\\\://admin\\\\:xxx@localhost\\\\:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0_%04d.mp4:master_pl_name=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.m3u8:master_pl_publish_rate=2:method=PUT]http://admin:xxx@localhost:8080/memfs/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf_output_0.m3u8|[f=flv]rtmp://localhost:1935/live/f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf.stream?token=foobar|[f=mpegts]srt://localhost:6000?mode=caller\u0026transtype=live\u0026streamid=f13d9ff4-8ac1-458d-8d61-6e7d8ad06faf,mode:publish,token:abc",
        "avstream": null,
        "bitrate_kbit": 0,
        "codec": "aac",
        "coder": "",
        "format": "tee",
        "fps": 0,
        "frame": 0,
        "id": "output_0",
        "index": 0,
        "layout": "stereo",
        "packet": 0,
        "pps": 0,
        "sampling_hz": 44100,
        "size_kb": 0,
        "stream": 1,
        "type": "audio"
      }
    ],
    "packet": 0,
    "q": 29,
    "size_kb": 0,
    "speed": 1,
    "time": 2488.84
  },
  "reconnect_seconds": -1,
  "runtime_seconds": 2489
}

Example: read the state of a process with the ID test: