Object Storage Features & Limitations
This page documents the features available on the Infomaniak Ceph Object Storage platform.
Feature Matrix
| Feature | Status | Notes |
|---|---|---|
| Object Lock | Supported | Version-level immutability. GOVERNANCE & COMPLIANCE modes supported. |
| Multipart Upload | Supported | Create, list, abort, complete. Minimum part: 5 MB (last part exempt), maximum part: 5GB, 10 000 parts max. |
| SSE-C (Customer-Provided Key) | Supported | AES256. Key + MD5 must be provided on every request. |
| SSE-S3 (Server-Side Encryption) | Not tested | AWS CLI rejects --server-side-encryption against RGW. Untested via raw HTTP. |
| Versioning | Supported | Enable, suspend, list versions, delete specific version. |
| Bucket Replication | Supported | Enable, disable, get configuration. Cross-region supported. |
| Object Tagging | Supported | Via x-amz-tagging on upload or put-object-tagging. Up to 10 tags. |
| User Metadata | Supported | x-amz-meta-* headers returned on HEAD/GET. Max 2 KB per object. |
| Lifecycle Policies | Supported | Expiration rules, prefix filtering, multipart cleanup. |
| CORS | Supported | PUT/GET bucket CORS configuration. |
| Requester Pays | Supported | Bucket-level requester-charges configuration. |
| Copy Object | Supported | Cross-bucket, cross-tenant (with permissions). Metadata replaced or copied. |
| S3 Select | Deprecated | Deprecated by AWS and being removed in Ceph. Do not use for new workloads. |
| Presigned URLs | Supported | Temporary access via query-string signature. |
| Public Buckets | Supported | Path-style required for anonymous access with Keystone multi-tenancy. |
| Bucket Policies (cross-tenant) | Supported | Principal: arn:aws:iam::<tenant>:root works. |
| Static Website Hosting | Not Supported | Will return MethodNotAllowed. |
| Event Notifications | Not Configured | Requires RGW-side AMQP/Kafka/HTTP endpoint. |
| SSE-KMS | Not Supported | No KMS integration. Use SSE-C. |
| Bucket Logging | Supported | "Lazy" , can take several hours to write logs. |
Multipart Upload
Multipart upload is the only way to upload objects larger than 5 GB. Ceph Object Storage supports the complete multipart workflow.
Step-by-step: upload a 12 MB file in 2 parts
Step 1 — Initiate the upload
aws s3api create-multipart-upload \
--bucket my-bucket \
--key data/large-file.dat \
--profile my-profile --region gva2
Output:
{
"UploadId": "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW",
"Bucket": "my-bucket",
"Key": "data/large-file.dat"
}
Store this UploadId. It is required for every subsequent operation.
Step 2a — Upload part 1 (5 MB minimum, last part exempt)
dd if=/dev/zero of=/tmp/part1.bin bs=1M count=5 status=none
aws s3api upload-part \
--bucket my-bucket --key data/large-file.dat \
--part-number 1 --body /tmp/part1.bin \
--upload-id "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW" \
--profile my-profile --region gva2 \
--query ETag --output text
# "5f363e0e58a95f06cbe9bbc662c5dfb6"
Step 2b — Upload part 2 (7 MB)
dd if=/dev/zero of=/tmp/part2.bin bs=1M count=7 status=none
aws s3api upload-part \
--bucket my-bucket --key data/large-file.dat \
--part-number 2 --body /tmp/part2.bin \
--upload-id "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW" \
--profile my-profile --region gva2 \
--query ETag --output text
# ETag of part 2
Step 3 — Complete the multipart upload
You MUST provide ALL parts with their exact ETags and part numbers:
aws s3api complete-multipart-upload \
--bucket my-bucket --key data/large-file.dat \
--upload-id "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW" \
--multipart-upload 'Parts=[{ETag="5f363e0e58a95f06cbe9bbc662c5dfb6",PartNumber=1},{ETag="ETAG_PART2_HERE",PartNumber=2}]' \
--profile my-profile --region gva2
Output:
{
"ETag": "\"81485c0e873d222199469076b60f30e9-2\"",
"Location": "s3.gva2.infomaniak.cloud/tenant:my-bucket/data/large-file.dat",
"Bucket": "my-bucket",
"Key": "data/large-file.dat"
}
The ETag ends with -2 which means it is a multipart object with 2 parts.
Inspect an active multipart upload
List uploads in progress for this bucket:
aws s3api list-multipart-uploads \
--bucket my-bucket \
--profile my-profile --region gva2
Output (if an upload is active):
{
"Uploads": [
{
"UploadId": "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW",
"Key": "data/large-file.dat",
"Initiated": "2026-06-25T11:03:15+00:00"
}
]
}
List the parts already uploaded for a specific upload:
aws s3api list-parts \
--bucket my-bucket --key data/large-file.dat \
--upload-id "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW" \
--profile my-profile --region gva2
Output:
{
"Parts": [
{
"PartNumber": 1,
"LastModified": "2026-06-25T11:03:20+00:00",
"ETag": "\"5f363e0e58a95f06cbe9bbc662c5dfb6\"",
"Size": 5242880,
"ChecksumCRC64NVME": "vqf3hRLTlJw="
}
]
}
Abort an incomplete upload
If you do not complete an upload, it remains on the server and consumes storage. Abort it explicitly:
aws s3api abort-multipart-upload \
--bucket my-bucket --key data/large-file.dat \
--upload-id "2~TPPI1kyv_ZNI1VQqMSJ4orj45ycjhNW" \
--profile my-profile --region gva2
No output on success. After abort, the part data is permanently removed.
Multipart Upload Size Limits
Hard limit: ~48.8 TB per object
S3 multipart uploads have two fixed limits:
| Limit | Value | What it means |
|---|---|---|
| Maximum parts | 10 000 | You cannot upload more than 10 000 parts for a single object |
| Maximum part size | 5 GB | Each individual part cannot exceed 5 GB |
Maximum object size: 5 GB × 10 000 = ~48.8 TB
Objects larger than ~48 TB cannot be stored as a single S3 object. You must split them into multiple objects (e.g., dataset/part-001, dataset/part-002).
Practical examples
| Object size | Part size | Number of parts | Status |
|---|---|---|---|
| 10 TB | 5 GB | ~2 050 | Works |
| 40 TB | 5 GB | ~8 200 | Works |
| 48 TB | 5 GB | ~9 830 | Works (near limit) |
| 100 TB | 5 GB | 20 480 | Fails — exceeds 10 000 part limit |
Recommended tool: rclone
For files approaching the limit, rclone automatically optimizes multipart parameters:
rclone copy large-file.dat s3aas1:my-bucket/
rclone will use the largest possible parts (up to 5 GB) and warn if the file exceeds what can be uploaded as a single object.
Manual multipart upload
If using aws s3api directly for large files, ensure you stay under both limits. The minimum part size (except the last) is 5 MB.
# Example: Upload a 10 TB file with 5 GB parts (~2 048 parts)
# 1. Initiate
aws s3api create-multipart-upload --bucket my-bucket --key large-file.dat
# 2. Upload parts (5 GB each, last part can be smaller)
for i in $(seq 1 2048); do
aws s3api upload-part --bucket my-bucket --key large-file.dat \
--part-number $i --body part-$i.bin --upload-id "YOUR_UPLOAD_ID"
done
# 3. Complete
aws s3api complete-multipart-upload --bucket my-bucket --key large-file.dat \
--upload-id "YOUR_UPLOAD_ID" --multipart-upload file://parts.json
Do NOT exceed 10 000 parts
Attempting to complete a multipart upload with more than 10 000 parts will fail with an error. Always verify: total_parts <= 10000 and each_part <= 5 GB.
Automated multipart cleanup with lifecycle
See the lifecycle section below for automatically removing incomplete multipart uploads older than N days.
Server-Side Encryption (SSE-C)
SSE-C lets you provide your own AES-256 encryption key. Ceph encrypts the object but does not store the key. You must provide the same key on every GET/HEAD.
Generate a key
# Generate 32 bytes (256 bits) as Base64
export SSE_KEY=$(openssl rand -base64 32)
# Compute MD5 of the raw key bytes (not the Base64 string)
export SSE_KEY_MD5=$(echo -n "$SSE_KEY" | openssl md5 -binary | base64)
echo "Key (keep secret): $SSE_KEY"
echo "MD5 (sent in headers): $SSE_KEY_MD5"
Example output:
Key (keep secret): 2R6ks18Jg+ZAWDHqtviNX5ZGwVhcr7evIS2gU9Iesp8=
MD5 (sent in headers): 5OA/MjuqXDtMP04vL9rq/Q==
Important: The MD5 is sent to the server via x-amz-server-side-encryption-customer-key-MD5 so the server can validate it received the correct key.
Upload with SSE-C
aws s3api put-object \
--bucket my-bucket \
--key sensitive-report.pdf \
--body ./sensitive-report.pdf \
--sse-customer-algorithm AES256 \
--sse-customer-key "$SSE_KEY" \
--sse-customer-key-md5 "$SSE_KEY_MD5" \
--profile my-profile --region gva2
Output:
{
"ETag": "\"6f5902ac237024bdd0c176cb93063dc4\"",
"SSECustomerAlgorithm": "AES256",
"SSECustomerKeyMD5": "t8xKGB5x0owGs4iEFHZ+qg=="
}
Notice SSECustomerAlgorithm and SSECustomerKeyMD5 in the response.
Download with SSE-C
You MUST provide the same key, or the GET will fail:
aws s3api get-object \
--bucket my-bucket \
--key sensitive-report.pdf \
--sse-customer-algorithm AES256 \
--sse-customer-key "$SSE_KEY" \
--sse-customer-key-md5 "$SSE_KEY_MD5" \
./downloaded-report.pdf \
--profile my-profile --region gva2
Output:
{
"AcceptRanges": "bytes",
"ContentLength": 1337,
"ContentType": "application/octet-stream",
"ETag": "\"6f5902ac237024bdd0c176cb93063dc4\"",
"LastModified": "2026-06-25T11:03:15+00:00",
"SSECustomerAlgorithm": "AES256",
"SSECustomerKeyMD5": "t8xKGB5x0owGs4iEFHZ+qg=="
}
HEAD without key fails
aws s3api head-object \
--bucket my-bucket \
--key sensitive-report.pdf \
--profile my-profile --region gva2
Error:
An error occurred (400) when calling the HeadObject operation: Bad Request
SSE-C objects cannot be read without the key. The server returns 400 Bad Request because it needs the key to decrypt and verify the object.
Note: SSE-S3 (Ceph-managed encryption) could not be validated because the AWS CLI rejects --server-side-encryption against RGW endpoints. Use SSE-C as a reliable alternative.
Versioning
When enabled, every PUT on an existing key creates a new version. Old versions are preserved until explicitly deleted.
Enable versioning on a bucket
aws s3api put-bucket-versioning \
--bucket my-bucket \
--versioning-configuration Status=Enabled \
--profile my-profile --region gva2
Get the current status:
aws s3api get-bucket-versioning \
--bucket my-bucket \
--profile my-profile --region gva2
Output:
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
Upload two versions of the same file
# Version 1 — upload original content
echo "original content v1" > /tmp/file.txt
aws s3 cp /tmp/file.txt s3://my-bucket/versioned.txt \
--profile my-profile --region gva2
# upload: ./file.txt to s3://my-bucket/versioned.txt
# Version 2 — overwrite with new content
echo "updated content v2" > /tmp/file.txt
aws s3 cp /tmp/file.txt s3://my-bucket/versioned.txt \
--profile my-profile --region gva2
# upload: ./file.txt to s3://my-bucket/versioned.txt
List versions
aws s3api list-object-versions \
--bucket my-bucket \
--prefix versioned.txt \
--profile my-profile --region gva2
Output:
{
"Versions": [
{
"ETag": "\"9d4d14c9f2d54d1249e6ac856dbdf9ad\"",
"Size": 19,
"StorageClass": "STANDARD",
"Key": "versioned.txt",
"VersionId": "iD2IkNeb0v5UA4swLlf-ZBBld1Pad-e",
"IsLatest": true,
"LastModified": "2026-06-25T11:03:28+00:00",
"Owner": {
"DisplayName": "PCP-V2ZD4G3",
"ID": "ecff9c4d00d5497f8cd7ee9482c652ad$ecff9c4d00d5497f8cd7ee9482c652ad"
}
},
{
"ETag": "\"7f67cac5a4edd578dea1c95387bad47c\"",
"Size": 18,
"Key": "versioned.txt",
"VersionId": "meCZ5HWVF2fb0pPIIQ2ChxlPQYqDK2p",
"IsLatest": false,
"LastModified": "2026-06-25T11:03:27+00:00"
}
]
}
Key observations:
VersionIdis unique per uploadIsLatest: trueon the most recentIsLatest: falseon older versions
Read an old version explicitly
aws s3api get-object \
--bucket my-bucket \
--key versioned.txt \
--version-id meCZ5HWVF2fb0pPIIQ2ChxlPQYqDK2p \
/tmp/old-version.txt \
--profile my-profile --region gva2
cat /tmp/old-version.txt
# original content v1
Delete a specific version (permanent)
aws s3api delete-object \
--bucket my-bucket \
--key versioned.txt \
--version-id meCZ5HWVF2fb0pPIIQ2ChxlPQYqDK2p \
--profile my-profile --region gva2
Output:
{
"VersionId": "meCZ5HWVF2fb0pPIIQ2ChxlPQYqDK2p"
}
The version is permanently removed. No delete marker is created.
Warning: aws s3 rm s3://my-bucket/versioned.txt (without --version-id) creates a DeleteMarker, not a real deletion. The file remains in old versions.
Suspend versioning
aws s3api put-bucket-versioning \
--bucket my-bucket \
--versioning-configuration Status=Suspended \
--profile my-profile --region gva2
After suspension, new uploads overwrite the current version without creating new versions. Existing versions are preserved.
Bucket Replication
Bucket replication automatically copies bucket content to another datacenter. In case of Disaster Recovery scenario such as primary Datacenter down, Infomaniak can switch the S3 endpoints to the secondary zone, no action needed for customners, replicated objects will be available as nothing happened.
Prerequisites
- The source bucket must have versioning enabled.
- The target bucket must exist and have versioning enabled.
- A bucket replication policy JSON file is required.
Enable replication
Create replication.json:
{
"Role": "arn:aws:iam::account-id:role/replication-role",
"Rules": [
{
"Status": "Enabled",
"Priority": 1,
"Filter": {},
"Destination": {
"Bucket": "arn:aws:s3:::my-bucket",
"StorageClass": "STANDARD"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
Note
DeleteMarkerReplication is not implemented in Ceph. Delete markers are always replicated regardless of this setting.
Apply it:
aws s3api put-bucket-replication \
--bucket my-bucket \
--replication-configuration file://replication.json \
--profile my-profile --region gva2
Verify replication status
aws s3api get-bucket-replication \
--bucket my-bucket \
--profile my-profile --region gva2
Output:
{
"ReplicationConfiguration": {
"Role": "",
"Rules": [
{
"ID": "",
"Priority": 1,
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3::<tenant-id>:my-bucket",
"StorageClass": "STANDARD"
}
}
]
}
}
Disable replication
aws s3api delete-bucket-replication \
--bucket my-bucket \
--profile my-profile --region gva2
No output on success. Existing replicated objects remain in the destination bucket.
Object Tagging
Tags are key-value strings attached to objects. Useful for classification, cost allocation, and filtering. Maximum 10 tags per object.
Add tags during upload
aws s3api put-object \
--bucket my-bucket \
--key report.pdf \
--body ./annual-report.pdf \
--tagging "env=prod&team=finance&year=2026" \
--profile my-profile --region gva2
Add or overwrite tags on an existing object
aws s3api put-object-tagging \
--bucket my-bucket \
--key report.pdf \
--tagging '{
"TagSet": [
{"Key": "env", "Value": "prod"},
{"Key": "team", "Value": "finance"},
{"Key": "retention", "Value": "7-years"}
]
}' \
--profile my-profile --region gva2
Read tags
aws s3api get-object-tagging \
--bucket my-bucket \
--key report.pdf \
--profile my-profile --region gva2
Output:
{
"TagSet": [
{
"Key": "env",
"Value": "prod"
},
{
"Key": "team",
"Value": "finance"
},
{
"Key": "retention",
"Value": "7-years"
}
]
}
Delete all tags
aws s3api delete-object-tagging \
--bucket my-bucket \
--key report.pdf \
--profile my-profile --region gva2
No output. Subsequent get-object-tagging returns an empty TagSet.
User Metadata
User metadata is a set of custom key-value headers stored alongside the object and returned on every HEAD/GET.
Upload with metadata
aws s3api put-object \
--bucket my-bucket \
--key dataset-2026.csv \
--body ./sales.csv \
--metadata "project=alpha,owner=taylor,department=engineering,classification=internal" \
--profile my-profile --region gva2
Read metadata via HEAD
aws s3api head-object \
--bucket my-bucket \
--key dataset-2026.csv \
--profile my-profile --region gva2
Output:
{
"ContentLength": 1337,
"ContentType": "application/octet-stream",
"ETag": "\"d6eb32081c822ed572b70567826d9d9d\"",
"LastModified": "2026-06-25T11:03:15+00:00",
"Metadata": {
"x-amz-meta-classification": "internal",
"x-amz-meta-department": "engineering",
"x-amz-meta-owner": "taylor",
"x-amz-meta-project": "alpha"
}
}
AWS prefixes all user metadata keys with x-amz-meta- automatically.
Immutable metadata
Metadata is immutable for a given version. To change metadata on an existing object without uploading new data, copy the object to itself and replace metadata:
aws s3api copy-object \
--copy-source "my-bucket/dataset-2026.csv" \
--bucket my-bucket \
--key dataset-2026.csv \
--metadata-directive REPLACE \
--metadata "project=beta,owner=taylor,updated=2026-06-25" \
--profile my-profile --region gva2
This creates a new version (if versioning is enabled) or overwrites the object.
Lifecycle Policies
Automate expiration, transition, and cleanup based on object age or prefix.
Rule: delete objects in logs/ after 30 days
Create a file lifecycle.json:
{
"Rules": [
{
"ID": "expire-logs-after-30-days",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled",
"Expiration": {
"Days": 30
}
}
]
}
Apply it:
aws s3api put-bucket-lifecycle-configuration \
--bucket my-bucket \
--lifecycle-configuration file://lifecycle.json \
--profile my-profile --region gva2
Rule: clean up incomplete multipart uploads after 7 days
Create cleanup.json:
{
"Rules": [
{
"ID": "cleanup-incomplete-uploads",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
}
]
}
Apply it:
aws s3api put-bucket-lifecycle-configuration \
--bucket my-bucket \
--lifecycle-configuration file://cleanup.json \
--profile my-profile --region gva2
Read current rules
aws s3api get-bucket-lifecycle-configuration \
--bucket my-bucket \
--profile my-profile --region gva2
Output:
{
"Rules": [
{
"Expiration": {
"Days": 30
},
"ID": "expire-logs-after-30-days",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled"
}
]
}
Delete all lifecycle rules
Send an empty configuration:
aws s3api put-bucket-lifecycle-configuration \
--bucket my-bucket \
--lifecycle-configuration '{"Rules": []}' \
--profile my-profile --region gva2
CORS (Cross-Origin Resource Sharing)
Allow browsers to read/write your bucket from other origins. Required for direct browser-to-S3 uploads and JavaScript apps.
Configuration file cors.json
{
"CORSRules": [
{
"AllowedOrigins": [
"https://example.com",
"https://app.example.com"
],
"AllowedMethods": [
"GET",
"HEAD",
"PUT",
"POST"
],
"AllowedHeaders": [
"*"
],
"MaxAgeSeconds": 3000
}
]
}
If you want to allow any origin (for development only, dangerous in production):
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"MaxAgeSeconds": 3000
}
]
}
Apply and verify
aws s3api put-bucket-cors \
--bucket my-bucket \
--cors-configuration file://cors.json \
--profile my-profile --region gva2
aws s3api get-bucket-cors \
--bucket my-bucket \
--profile my-profile --region gva2
Output:
{
"CORSRules": [
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedOrigins": ["https://example.com"],
"MaxAgeSeconds": 3000
}
]
}
Test with curl (preflight)
curl -I -X OPTIONS \
-H "Origin: https://example.com" \
-H "Access-Control-Request-Method: GET" \
"https://s3.gva2.infomaniak.cloud/my-bucket/test.txt"
Expected headers in response:
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Methods: GET, HEAD
Access-Control-Max-Age: 3000
Delete CORS
aws s3api delete-bucket-cors \
--bucket my-bucket \
--profile my-profile --region gva2
Requester Pays
By default, the bucket owner pays for bandwidth. Enable Requester Pays to make the caller tenant pay instead. This is the AWS S3 mechanism for shifting egress costs from the bucket owner to the requester.
Enable Requester Pays
aws s3api put-bucket-request-payment \
--bucket my-bucket \
--request-payment-configuration '{"Payer": "Requester"}' \
--profile my-profile --region gva2
Verify
aws s3api get-bucket-request-payment \
--bucket my-bucket \
--profile my-profile --region gva2
Output:
{
"Payer": "Requester"
}
How access changes with Requester Pays
Bucket owner: No change. Continues to access normally without any additional header.
Requester (another tenant or anonymous): Must include x-amz-request-payer: requester in every request. Without this header, requests are rejected.
# Requester must add --request-payer requester
curl -I -H "x-amz-request-payer: requester" \
"https://s3.gva2.infomaniak.cloud/<tenant>:<bucket>/<object>"
Important: Requester Pays does not make a private bucket public. The requester still needs read permission via a bucket policy or ACL. The x-amz-request-payer header only affects who is billed for the transfer.
Disable Requester Pays
aws s3api put-bucket-request-payment \
--bucket my-bucket \
--request-payment-configuration '{"Payer": "BucketOwner"}' \
--profile my-profile --region gva2
Infomaniak billing model
Storage costs vs traffic costs
On Infomaniak Ceph Object Storage:
- Storage is billed to the bucket owner (the tenant that owns the bucket)
- There are currently no traffic/egress fees on Infomaniak Ceph Object Storage
- Requester Pays is implemented and functional at the API level
Because Infomaniak does not charge for traffic, Requester Pays has no practical billing effect in the current pricing model. The feature is available for API compatibility and future use, but enabling it will not shift any actual costs to requesters.
If traffic-based billing is introduced in the future, Requester Pays will become relevant for cross-tenant shared datasets.
Copy Object
Copy objects within a bucket, across buckets, or across tenants.
Simple same-bucket copy
aws s3api copy-object \
--copy-source "my-bucket/source.txt" \
--bucket my-bucket \
--key backup/source.txt \
--profile my-profile --region gva2
Output:
{
"CopyObjectResult": {
"ETag": "\"d6eb32081c822ed572b70567826d9d9d\"",
"LastModified": "2026-06-25T11:03:45+00:00"
}
}
Cross-bucket copy
Requires read access on source and write access on destination.
aws s3api copy-object \
--copy-source "source-bucket/backup.zip" \
--bucket destination-bucket \
--key imported/backup.zip \
--profile my-profile --region gva2
Copy a specific version
aws s3api copy-object \
--copy-source "my-bucket/document.pdf?version-id=iD2IkNeb0v5UA4swLlf-ZBBld1Pad-e" \
--bucket my-bucket \
--key document-restore.pdf \
--profile my-profile --region gva2
Copy and replace metadata
aws s3api copy-object \
--copy-source "my-bucket/old-report.pdf" \
--bucket my-bucket \
--key new-report.pdf \
--metadata-directive REPLACE \
--metadata "version=2,status=final,owner=alice" \
--profile my-profile --region gva2
Copy with preserved metadata (default)
aws s3api copy-object \
--copy-source "my-bucket/source.txt" \
--bucket my-bucket \
--key dest.txt \
--metadata-directive COPY \
--profile my-profile --region gva2
If --metadata-directive is omitted, metadata is copied by default.
Bucket Logging
Bucket logging provides logging for the requests that are made to a bucket. The log data can be used to monitor bucket activity, detect unauthorized access, get insights into bucket usage, and use the logs as a journal for bucket changes.
Note : The log bucket cannot have logging enabled on it. The log bucket cannot have any encryption set on it. The log bucket must not have RequestPayer enabled. The log bucket may have object lock enabled with default retention period.
Step-by-step: enable bucket logging on 2 different buckets
** Step 1 - Create two buckets
aws s3 mb s3://my-bucket1
aws s3 mb s3://my-bucket2
** Step 2 - Create the log bucket
aws s3 mb s3://my-logging-bucket
** Step 3 - Allow the sources buckets ( my-bucket[1-2] ) to put logs on the log bucket
- create and apply the bucket policy document:
cat > allow-logging.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowLoggingFromSourceBucket",
"Effect": "Allow",
"Principal": {
"Service": "logging.s3.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": “arn:aws:s3::<tenant-id>:my-logging-bucket/*”,
"Condition": {
"StringEquals": {
"aws:SourceAccount": “<tenant-id>$<tenant-id>“,
"aws:SourceArn": [
“arn:aws:s3::<tenant-id>:my-bucket1”,
“arn:aws:s3::<tenant-id>:my-bucket2”
]
}
}
}
]
}
EOF
aws s3api put-bucket-policy --bucket=my-logging-bucket --policy file://allow-logging.json
- Enable bucket logging on the source buckets :
cat > my-bucket1-logging.json <<EOF
{
"LoggingEnabled": {
"TargetBucket": "my-logging-bucket",
"TargetPrefix": "my-bucket1/"
}
}
aws s3api put-bucket-logging --bucket my-bucket1 --bucket-logging-status file://my-bucket1-logging.json
Repeat the step 2-b for the my-bucket2
** Inspect the applied policies :
Check the bucket policy on the my-logging-bucket:
aws s3api get-bucket-policy --bucket=my-logging-bucket
Example Output:
{
"Policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"AllowLoggingFromSourceBucket\",\n \"Effect\": \"Allow\",\n \"Principal\": {\n \"Service\": \"logging.s3.amazonaws.com\"\n },\n \"Action\": \"s3:PutObject\",\n \"Resource\": \"arn:aws:s3::2c718fb4ecb440e7917bc9312bb99988:my-logging-bucket/*\",\n \"Condition\": {\n \"StringEquals\": {\n \"aws:SourceAccount\": \"2c718fb4ecb440e7917bc9312bb99988$2c718fb4ecb440e7917bc9312bb99988\",\n \"aws:SourceArn\": [\n \"arn:aws:s3::2c718fb4ecb440e7917bc9312bb99988:my-bucket-1\",\n \"arn:aws:s3::2c718fb4ecb440e7917bc9312bb99988:my-bucket-2\"\n ]\n }\n }\n }\n ]\n}\n"
}
Check the applied bucket logging policy :
aws s3api get-bucket-logging —bucket=my-bucket-1
Example Output:
{
"LoggingEnabled": {
"TargetBucket": "my-logging-bucket",
"TargetPrefix": "my-bucket-1/",
"TargetObjectKeyFormat": {
"SimplePrefix": {}
}
}
}
** Deactivate bucket logging on an enabled logging bucket:
aws s3api put-bucket-logging --bucket my-bucket-1 --bucket-logging-status ‘{}'
- Log Format:
The S3 access logs are stored with the following format :
<Targetprefix>/<year-month-day-hour-minute-second>-<16 bytes unique-id>
Note : Logging to the bucket may take several hours after the operation to the log is been written to the target bucket. To accelerate that, you can flush all logging objects on a given source bucket to log them . Flushing will happen automatically when logging is deactivated on a bucket, or its logging configuration is changed, or the bucket is deleted.
Known Limitations
AWS CLI Compatibility Issues
Some AWS CLI commands fail against RGW before the request is even sent:
| Operation | CLI Error | Workaround |
|---|---|---|
aws s3 sync |
XML parsing errors | Use s3cmd sync or rclone |
aws s3 cp --server-side-encryption |
ParamValidation: Unknown options |
Use SSE-C (--sse-customer-*) instead |
aws s3api put-bucket-website |
NoneType is not iterable |
Not supported on cluster |
aws s3 ls on large buckets |
Slow / timeout | Use s3cmd or rclone |
Features Not Supported on Tested Cluster
| Feature | Status | Alternative |
|---|---|---|
| Static Website Hosting | Returns MethodNotAllowed |
Use a CDN or reverse proxy (e.g., Nginx with proxy_pass) |
| Event Notifications | No SNS/AMQP/Kafka configured | Poll with ListObjectsV2 or use a Lambda-like external worker |
| SSE-KMS | No KMS integration | Use SSE-C with your own keys |