S3

Last updated 2 years ago

Was this helpful?

S3

S3 filesystems are only accessible via HTTP their configured mountpoint. Use the POST or PUT method with the path to that file to (over-)write a file. The body of the request contains the contents of the file. No particular encoding or Content-Type is required. The file can then be downloaded from the same path.

This filesystem is practical rarely changing data (e.g. VOD) for long term storage.

On this page and in the examples we assume that a S3 storage with the name aws is mounted on /awsfs.

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -X PUT -d @somefile.txt

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -o file.txt

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -X DELETE

The returned Content-Type is based solely on the file extension. For a list of known mime-types and their extensions see in the configuration.

Access protection

It is strongly recommended to enable a username/password (HTTP Basic-Auth) protection for any PUT/POST and DELETE operations on /memfs. GET operations are not protected.

By default HTTP Basic-Auth is not enabled.

API

The contents of the S3 filesystem mounted on /awsfs are also accessible via the API in the same way as described above, but with the same protection as the API (see ) for all operations. It is also possible to list all files that are currently in the filesystem.

Create, Update

Example:

echo 'test' > example.txt && \
curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -d @example.txt \
   -X PUT

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

client.v3_fs_put_file(
    name: "aws",
    path: "example.txt",
    data: b"test"
)

import (
    "strings"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

data := strings.NewReader("test")
err := client.FilesystemAddFile("aws", "/example.txt", data)

The contents for the upload has to be provided as an io.Reader.

After the successful upload the file is available at /awsfs/example.txt and /api/v3/fs/aws/example.txt.

Description:

Read

List all files

Listing all currently stored files is done by calling /api/v3/fs/aws. It also accepts the query parameters pattern, sort (name, size, or lastmod) and order (asc or desc). If none of the parameters are given, all files will be listed sorted by their last modification time in ascending order.

Example:

curl "http://127.0.0.1:8080/api/v3/fs/aws?sort=name&order=asc" \
   -X GET

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

core_memfs_list = client.v3_fs_get_file_list(name="aws")
print(core_memfs_list)

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

files, err := client.FilesystemList("aws", "/*.", coreclient.SORT_NAME, coreclient.ORDER_ASC)

for _, file := range files {
    fmt.Printf("%+v\n", file)
}

Description:

Download a file

For downloading a file you have to specify the complete path and filename. The Content-Type will always be application/data.

Example:

curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -X GET

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

core_memfs_file = client.v3_fs_get_file(
    name="aws",
    path="example.txt"
)
print(core_memfs_file)

import (
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

data, err := client.FilesystemGetFile("aws", "/example.txt")
defer data.Close()

The returned data is an io.ReadCloser.

Description:

Delete

For deleting a file you have to specify the complete path and filename.

Example:

curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -X DELETE

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

client.v3_fs_delete_file(
    name="aws",
    path="example.txt"
)

import (
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

err := client.FilesystemDeleteFile("aws", "/example.txt")

Description:

Configuration

PreviousIn-memory NextMetrics

Last updated 2 years ago

Was this helpful?

This filesystem is practical rarely changing data (e.g. VOD) for long term storage.

On this page and in the examples we assume that a S3 storage with the name aws is mounted on /awsfs.

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -X PUT -d @somefile.txt

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -o file.txt

curl http://127.0.0.1:8080/awsfs/path/to/a/file.txt -X DELETE

The returned Content-Type is based solely on the file extension. For a list of known mime-types and their extensions see in the configuration.

Access protection

It is strongly recommended to enable a username/password (HTTP Basic-Auth) protection for any PUT/POST and DELETE operations on /memfs. GET operations are not protected.

By default HTTP Basic-Auth is not enabled.

API

Create, Update

Example:

echo 'test' > example.txt && \
curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -d @example.txt \
   -X PUT

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

client.v3_fs_put_file(
    name: "aws",
    path: "example.txt",
    data: b"test"
)

import (
    "strings"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

data := strings.NewReader("test")
err := client.FilesystemAddFile("aws", "/example.txt", data)

The contents for the upload has to be provided as an io.Reader.

After the successful upload the file is available at /awsfs/example.txt and /api/v3/fs/aws/example.txt.

Description:

Read

List all files

With the pattern parameter you can filter the list based on a , with the addition of the ** placeholder to include multiple subdirectories, e.g. listing all .ts file in the root directory has the pattern /*.ts, listing all .ts file in the whole filesystem has the pattern /**.ts.

Example:

curl "http://127.0.0.1:8080/api/v3/fs/aws?sort=name&order=asc" \
   -X GET

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

core_memfs_list = client.v3_fs_get_file_list(name="aws")
print(core_memfs_list)

import (
    "fmt"
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

files, err := client.FilesystemList("aws", "/*.", coreclient.SORT_NAME, coreclient.ORDER_ASC)

for _, file := range files {
    fmt.Printf("%+v\n", file)
}

Description:

Download a file

For downloading a file you have to specify the complete path and filename. The Content-Type will always be application/data.

Example:

curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -X GET

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

core_memfs_file = client.v3_fs_get_file(
    name="aws",
    path="example.txt"
)
print(core_memfs_file)

import (
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

data, err := client.FilesystemGetFile("aws", "/example.txt")
defer data.Close()

The returned data is an io.ReadCloser.

Description:

Delete

For deleting a file you have to specify the complete path and filename.

Example:

curl http://127.0.0.1:8080/api/v3/fs/aws/example.txt \
   -X DELETE

from core_client import Client

client = Client(base_url="http://127.0.0.1:8080")
client.login()

client.v3_fs_delete_file(
    name="aws",
    path="example.txt"
)

import (
    "github.com/datarhei/core-client-go/v16"
)

client, _ := coreclient.New(coreclient.Config{
    Address: "https://127.0.0.1:8080",
})

err := client.FilesystemDeleteFile("aws", "/example.txt")

Description:

Configuration

List all files on a filesystem

get

List all files on a filesystem. The listing can be ordered by name, size, or date of last modification in ascending or descending order.

Authorizations

Path parameters

namestringRequired

Name of the filesystem

Query parameters

globstringOptional

glob pattern for file names

sortstringOptional

none, name, size, lastmod

orderstringOptional

asc, desc

Responses

200

application/json

get

GET /v3/fs/{name} HTTP/1.1
Host: api
Authorization: YOUR_API_KEY
Accept: */*

200

[
  {
    "last_modified": 1,
    "name": "text",
    "size_bytes": 1
  }
]

Fetch a file from a filesystem

get

Fetch a file from a filesystem

Authorizations

Path parameters

namestringRequired

Name of the filesystem

pathstringRequired

Path to file

Responses

200

Responsestring

301

Moved Permanently

404

Not Found

get

GET /v3/fs/{name}/{path} HTTP/1.1
Host: api
Authorization: YOUR_API_KEY
Accept: */*

text

Remove a file from a filesystem

delete

Remove a file from a filesystem

Authorizations

Path parameters

namestringRequired

Name of the filesystem

pathstringRequired

Path to file

Responses

200

text/plain

Responsestring

404

Not Found

text/plain

delete

DELETE /v3/fs/{name}/{path} HTTP/1.1
Host: api
Authorization: YOUR_API_KEY
Accept: */*

text

Add a file to a filesystem

put

Writes or overwrites a file on a filesystem

Authorizations

Path parameters

namestringRequired

Name of the filesystem

pathstringRequired

Path to file

Body

integer[]Optional

Responses

201

Created

Responsestring

204

No Content

507

Insufficient Storage

put

PUT /v3/fs/{name}/{path} HTTP/1.1
Host: api
Authorization: YOUR_API_KEY
Content-Type: application/data
Accept: */*
Content-Length: 3

[
  1
]

text