Introduction

Object Storage offers a scalable storage service which can be used by applications using HTTP or the S3 protocol.
It is ideal for storing unstructured data (called "blob" or "objects") such as documents, images, large binary files...

The Infomaniak Object Stores are built upon Swift.

Main features

Guarantees data availability, durability and consistency
Files can be sent using the dashboard or via an API
Unlimited storage capacity
Containers are private by default but can be public

Technical details

Files are stored in containers
The name of each container must be unique in the project
The upload is limited to :
- 5Gb per file via the dashboard
- 50Tb via API (multi-threading DLO/SLO)
A successful file upload returns an HTTP 200 code

Display capabilities

To know enabled plugins (middlewares) on our cluster, you can use:

swift capabilities

Using Swift Object Storage

CLI curl

Some commands are not available through the OpenStackClient. We will install the Swift Client.

pip install python3-swiftclient

Create a container:

I'll create a container named public:

openstack container create public

+---------------------------------------+-----------+------------------------------------+
| account                               | container | x-trans-id                         |
+---------------------------------------+-----------+------------------------------------+
| AUTH_d1440xxxxxxxxxxxxxxxxxxxxxxxxxxx | public    | tx59c2735e9bfd4a5181079-006048ddc8 |
+---------------------------------------+-----------+------------------------------------+

List your containers:

openstack container list

+--------+
| Name   |
+--------+
| public |
+--------+

Store an object:

openstack object create --name 'eBooks/Cuisine/Les Crêpes du Monde.pdf' public '~/Documents/Ebooks/Cuisine/Les Crêpes du Monde.pdf'

+----------------------------------------+-----------+----------------------------------+
| object                                 | container | etag                             |
+----------------------------------------+-----------+----------------------------------+
| eBooks/Cuisine/Les Crêpes du Monde.pdf | public    | a87f8e432d5c0c163d575a7fde411be8 |
+----------------------------------------+-----------+----------------------------------+

The --name option override the final object name.
By default, it uses your local object name.

List your objects:

openstack object list public

+----------------------------------------+
| Name                                   |
+----------------------------------------+
| eBooks/Cuisine/Les Crêpes du Monde.pdf |
+----------------------------------------+

Delete an object:

openstack object delete public 'eBooks/Cuisine/Les Crêpes du Monde.pdf'

Get an authentication token with switf client:

swift auth

or using curl, create a script called get_token.sh with the following content:

#!/bin/sh

TMPFILE=`mktemp`
JSONFILE=`mktemp`

cat >${JSONFILE} <<EOF
{
  "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "domain": {"name": "${OS_USER_DOMAIN_NAME}"},
          "name": "${OS_USERNAME}",
          "password": "${OS_PASSWORD}"
        }
      }
    },
    "scope": {
      "project": {
        "domain": {"name": "${OS_PROJECT_DOMAIN_NAME}"},
        "name": "${OS_PROJECT_NAME}"
      }
    }
  }
}
EOF

curl -si  \
  -H "Content-Type: application/json" \
  -o ${TMPFILE} \
  -d @${JSONFILE} \
${OS_AUTH_URL}/auth/tokens 2>/dev/null

tail -1 ${TMPFILE} | jq
grep -i "x-subject-token" ${TMPFILE}

rm -f ${TMPFILE} ${JSONFILE}

Once you have a the access token, you can operate with the Object Storage:

Create a container:

curl -i -X PUT -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/mycontainer

HTTP/1.1 202 Accepted
content-type: text/html; charset=UTF-8
content-length: 76
x-trans-id: tx1d77854fae534495a2a94-00655c8b2d
x-openstack-request-id: tx1d77854fae534495a2a94-00655c8b2d
date: Tue, 21 Nov 2023 10:49:17 GMT
strict-transport-security: max-age=63072000

<html><h1>Accepted</h1><p>The request is accepted for processing.</p></html>

Upload an object:

curl -i -T my_object -X PUT -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/mycontainer/my_object

Download an object:

curl -S -X GET -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/mycontainer/my_object -O

List containers:

curl -S -X GET -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/

List objects in a container:

curl -S -X GET -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/mycontainer/

Delete an object:

curl -S -X DELETE -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/mycontainer/myobject

Delete a container (must be empty):

curl -S -X DELETE -H "X-Auth-Token: $OS_AUTH_TOKEN" ${OS_STORAGE_URL}/mycontainer

Object versioning

Object Versioning is used to track changes to objects and to enable recovery of specific versions of an object.

Creating a bucket with versioning enabled

CLI curl

swift post -H "X-Versions-Enabled: true" mybucket

curl -i -XPUT -H "X-Auth-Token: ${OS_AUTH_TOKEN}" -H "X-Versions-Enabled: true" ${OS_STORAGE_URL}/mybucket

From now on, each uploaded object will be versioned with an ID. This ID is the timestamp of the upload.

swift upload mybucket object1
swift upload mybucket object1

Listing the objects and available versions

swift list mybucket --versions

183 2023-05-26 07:19:27 1685085567.70333 application/octet-stream object1
183 2023-05-26 07:19:22 1685085562.07717 application/octet-stream object1

Downloading a specific version of an object

swift download mybucket object1 --version-id 1685085562.07717

Deleting a specific version of an object

swift delete mybucket object1 --version-id 1685085567.70333

Note

Deleting a versioned object will only remove the latest version of the object. Older versions are therefore still available.

swift delete mybucket object1
swift list mybucket --versions

0 2023-05-26 07:21:00 1685085660.97534 application/x-deleted;swift_versions_deleted=1 object1
183 2023-05-26 07:20:54 1685085654.97972 application/octet-stream object1
183 2023-05-26 07:19:22 1685085562.07717 application/octet-stream object1

View your account statistics

openstack object store account show
swift list mybucket --versions

+------------+---------------------------------------+
| Field      | Value                                 |
+------------+---------------------------------------+
| Account    | AUTH_d7a1d0xxxxxxxxxxxxxxxxxxxxxxxxxx |
| Bytes      | 9149046                               |
| Containers | 1                                     |
| Objects    | 1                                     |
+------------+---------------------------------------+

Important considerations

An object store is not a filesystem

It cannot be stressed enough that your usage of the object store should reflect the proper use case, and not treat the storage like a traditional filesystem. There are two main restrictions to bear in mind when designing an application that uses an object store:

You cannot rename objects. Due to fact that the name of an object is one of the factors that determines where the object and its replicas are stored, renaming would require multiple copies of the data to be moved between physical storage devices. If you want to rename an object you must upload to the new location, or make a server side copy request to the new location, and then delete the original.
You cannot modify objects. Objects are stored in multiple locations and are checked for integrity based on the MD5 sum calculated during upload. In order to modify the contents of an object, the entire desired contents must be re-uploaded. In certain special cases it is possible to work around this restriction using large objects, but no general file-like access is available to modify a stored object.

Objects cannot be locked

There is no mechanism to perform a combination of reading the data/metadata from an object and writing an update to that data/metadata in an atomic way. Any user with access to a container could update the contents or metadata associated with an object at any time.

Workflows that assume that no updates have been made since the last read of an object should be discouraged. Enabling a workflow of this type requires an external object locking mechanism and/or cooperation between all clients accessing the data.