by: Linux/Unix Web

Installation:

sudo apt update
sudo apt install s3cmd -y

OR

sudo apt install python3-pip
sudo pip install s3cmd

Commands:

  • s3cmd –configure
    • Follow the steps to configure.
  • s3cmd ls
    • List existing buckets.
  • s3cmd ls s3://bucket_name/
    • List all objects in a bucket.
  • s3cmd ls s3://bucket_name/dir/
    • List all objects in a directory within a bucket.
  • s3cmd la
    • List all objects inside all buckets.
  • s3cmd put file.zip s3://bucket_name/
    • Upload a file.
  • s3cmd put –multipart-chunk-size-mb=100 file.zip s3://bucket_name/
    • Upload a file by splitting it into 100MB chunks. The final file will appear as one.
  • s3cmd get s3://bucket_name/file.zip
    • Download a file.
  • s3cmd del s3://bucket_name/file.zip
    • Delete a file.
  • s3cmd put -r /directory s3://bucket_name/
    • Upload a directory recursively.
  • s3cmd mv s3://bucket_name/origin/file.zip s3://bucket_name/destination/file.zip
    • Move a file.
  • s3cmd mv s3://bucket_name/origin/ s3://bucket_name/destination/
    • Move a directory recursively.
  • s3cmd sync . -r s3://bucket_name/
    • Sync a local directory with a bucket directory recursively.
  • s3cmd sync . –delete-removed s3://bucket_name/destination/
    • Sync a local directory with a bucket directory, deleting files that were removed from the origin.
  • s3cmd signurl s3://bucket_name/file.zip +600
    • Generate a public download URL valid for 10 minutes (600 seconds).
  • s3cmd signurl s3://bucket_name/file.zip $(date “+%s” -d “12/31/2030 23:59:59”)
    • Generate a public download URL valid until a specified date and time.
  • mysqldump –all-databases | gzip | s3cmd put – s3://bucketName/fileName.sql.gz
    • Pipe command output directly to an S3 bucket.

To throttle the upload speed (to 1 Mbps) on Linux, use the following command:

cat mybigfile | throttle -k 1024 | s3cmd put - s3://bucket_name/mybigfile

Each user’s configuration is saved in ~/.s3cfg.

BONUS

It is also possible to mount an S3 bucket in Ubuntu as a file system:

sudo apt install s3fs -y
echo ABGTKAHSICAKIAWDUVUL:JUllFbRAmydkgEz089/UZgm/VR1ShAZejOFiPan0 > /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs
mkdir /bucket-name
s3fs bucket-name /bucket-name

Note: use /etc/passwd-s3fs for system-wide access, or ~/.passwd-s3fs for a specific user.

To enable debug mode, use:

s3fs bucket-name /bucket-name -o dbglevel=info -f -o curldbg

To auto-mount the bucket on startup, add the following line to /etc/fstab:

s3fs#bucket-name /bucket-name fuse _netdev,allow_other,umask=227,uid=33,gid=33,use_cache=/root/cache 0 0

To make it compatible with Linode’s S3 service, use the following command:

s3fs bucket-name /bucket-name -o url="https://us-east-1.linodeobjects.com" -o endpoint=us-east-1 -o passwd_file=/etc/passwd-s3fs

For /etc/fstab:

s3fs#bucket-name /bucket-name fuse _netdev,allow_other,use_path_request_style,url=https://us-east-1.linodeobjects.com/,uid=userid,gid=groupid 0 0

Backup a whole directory (recursively), encrypt it, and push it to AWS S3:

tar -cvzf - directoryName | gpg -r "[email protected]" --yes --encrypt - | aws s3 cp - s3://bucketName/fileName.tar.gpg

BONUS

Cloudflare offers S3-compatible block storage called R2. Here is how the configuration file ~/.s3cfg should look:

[default]
host_base = <ACCOUNT_ID>.r2.cloudflarestorage.com
access_key = <ACCESS_KEY>
secret_key = <SECRET_KEY>
host_bucket = 
bucket_location = auto
enable_multipart = False

The AWS CLI tool can be used with other cloud providers, not just AWS. Below is an example configuration for connecting to Cloudflare R2, to be added to ~/.aws/credentials:

[default]
endpoint_url = https://<ACCOUNT_ID>.r2.cloudflarestorage.com
aws_access_key_id = <ACCESS_KEY>
aws_secret_access_key = <SECRET_KEY>
region = auto
output = json