Skip to main content

Using the S3 API

Use the S3 API when you want to script uploads, run backups, or use features that the web panel doesn't offer, such as versioning, Object Lock and lifecycle rules. Any S3-compatible tool works. This page sets up the three most common ones.

You need:

  • An access key ID and secret access key. See Enabling S3 access.
  • The endpoint of the region your bucket is in.

Endpoints​

RegionLocationEndpoint
DX1Dubai, UAEhttps://dxb.bbzs3.com
VIE2Vienna, Austriahttps://vie.bbzs3.com
FJ1Fujairah, UAEComing soon

The examples on this page use DX1. For Vienna, swap in the VIE2 endpoint.

Client settings​

SettingValueWhy
EndpointFrom the table abovePoints your tool at Bamboozle instead of AWS
Regionus-east-1Object Storage ignores the region name, but most tools need one. us-east-1 avoids tools sending a location setting when creating buckets.
Addressing stylePath-styleBucket name goes in the path: https://<endpoint>/<bucket>/<key>
SignatureVersion 4The default in all current tools. Version 2 is also accepted for old software.
ProtocolHTTPSAlways use HTTPS so keys and data are encrypted in transit

aws-cli​

Install aws-cli version 2, then create a profile so your Bamboozle settings stay separate from any AWS account:

cd ~
aws configure --profile bamboozle

Enter your access key ID and secret access key. For Default region name enter us-east-1, and for Default output format press Enter.

Then add the endpoint and S3 settings to the profile in ~/.aws/config:

[profile bamboozle]
region = us-east-1
endpoint_url = https://dxb.bbzs3.com
request_checksum_calculation = when_required
response_checksum_validation = when_required
s3 =
addressing_style = path

The two checksum lines stop newer aws-cli versions (2.23 and later) from adding checksum headers that S3-compatible storage may reject.

Test it:

cd ~
aws s3 ls --profile bamboozle

To avoid typing --profile bamboozle every time, run export AWS_PROFILE=bamboozle in your shell. With older aws-cli versions that don't read endpoint_url from the profile, add --endpoint-url https://dxb.bbzs3.com to each command.

Everyday commands​

cd ~
# Create a bucket
aws s3 mb s3://<bucket> --profile bamboozle

# List buckets, or the contents of a bucket
aws s3 ls --profile bamboozle
aws s3 ls s3://<bucket>/ --recursive --profile bamboozle

# Upload and download a file
aws s3 cp ./report.pdf s3://<bucket>/reports/report.pdf --profile bamboozle
aws s3 cp s3://<bucket>/reports/report.pdf ./report.pdf --profile bamboozle

# Copy only new and changed files from a folder (one way)
aws s3 sync ./website/ s3://<bucket>/website/ --profile bamboozle

# Delete a file, or everything under a prefix
aws s3 rm s3://<bucket>/reports/report.pdf --profile bamboozle
aws s3 rm s3://<bucket>/old-logs/ --recursive --profile bamboozle

# Share a file with a temporary link (valid for 1 hour)
aws s3 presign s3://<bucket>/reports/report.pdf --expires-in 3600 --profile bamboozle

aws s3 sync doesn't delete anything at the destination unless you add --delete. Run it with --dryrun first.

rclone​

rclone is a good fit for backups and large folder copies. Add this remote to ~/.config/rclone/rclone.conf, or create it interactively with rclone config (choose Amazon S3 compliant storage, then provider Other):

[bamboozle]
type = s3
provider = Other
access_key_id = <access-key>
secret_access_key = <secret-key>
endpoint = https://dxb.bbzs3.com
region = us-east-1
force_path_style = true
acl = private
cd ~
rclone lsd bamboozle:
rclone copy /srv/backups bamboozle:<bucket>/server01 --progress
rclone sync /srv/www bamboozle:<bucket>/www --dry-run

rclone sync deletes files at the destination that no longer exist at the source. Use rclone copy if you never want anything removed. To encrypt files before they leave your server, wrap this remote in an rclone crypt remote. See the backup target guide.

s3cmd​

For s3cmd, put this in ~/.s3cfg:

[default]
access_key = <access-key>
secret_key = <secret-key>
host_base = dxb.bbzs3.com
host_bucket = dxb.bbzs3.com
use_https = True
signature_v2 = False

Setting host_bucket to the plain endpoint makes s3cmd use path-style addressing.

cd ~
s3cmd ls
s3cmd put ./file.tar.gz s3://<bucket>/

SDKs​

Any AWS SDK works. Set the endpoint, the region us-east-1, and path-style addressing. For example, in Python with boto3:

import boto3
from botocore.config import Config

s3 = boto3.client(
"s3",
endpoint_url="https://dxb.bbzs3.com",
region_name="us-east-1",
aws_access_key_id="<access-key>",
aws_secret_access_key="<secret-key>",
config=Config(s3={"addressing_style": "path"}),
)
print(s3.list_buckets()["Buckets"])

Large files and multipart upload​

  • A single upload can be up to 5 GB. Larger files, up to 5 TB, must use multipart upload.
  • aws-cli, rclone, s3cmd and the SDKs switch to multipart automatically for large files, so you normally don't need to do anything.
  • If an upload is interrupted, its parts stay in the bucket and count as stored data until the upload is completed or aborted. Add a lifecycle rule that cleans them up automatically.

Troubleshooting​

ErrorLikely causeFix
SignatureDoesNotMatchWrong secret key, or a stray space when pasting itCopy the secret access key again from S3 > Access
RequestTimeTooSkewedThe clock on your machine is off by more than a few minutesTurn on time sync (NTP), e.g. timedatectl set-ntp true on Linux
NoSuchBucket or DNS errors for <bucket>.<endpoint>Your tool is using virtual-hosted-style addressingSwitch to path-style (see the settings above)
InvalidAccessKeyIdThe key was disabled or deleted, or you're using another region's endpointCheck the key under S3 > Access and the endpoint
AccessDeniedA bucket policy blocks the request, or the bucket belongs to another userCheck the bucket policy
Checksum or x-amz-content-sha256 errors with a new aws-cliNewer aws-cli versions send extra checksum headersAdd the two when_required lines to your profile (see above)
TLS or certificate errorsOutdated CA certificates on your machineUpdate your system's CA bundle. Don't turn off certificate checks.
Was this page helpful?