{"id":2108,"date":"2021-05-22T02:37:23","date_gmt":"2021-05-22T02:37:23","guid":{"rendered":"https:\/\/dft.wiki\/?p=2108"},"modified":"2026-06-08T22:48:41","modified_gmt":"2026-06-09T02:48:41","slug":"s3cmd-cheat-sheet-for-amazon-s3-object-storage","status":"publish","type":"post","link":"https:\/\/dft.wiki\/?p=2108","title":{"rendered":"S3CMD Cheat Sheet for Amazon S3 Object Storage"},"content":{"rendered":"<p>Installation:<\/p>\n<pre>sudo apt update\r\nsudo apt install s3cmd -y<\/pre>\n<p>OR<\/p>\n<pre>sudo apt install python3-pip\r\nsudo pip install s3cmd<\/pre>\n<p>Commands:<\/p>\n<ul>\n<li>s3cmd &#8211;configure\n<ul>\n<li>Follow the steps to configure.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>ls<\/strong>\n<ul>\n<li>List existing buckets.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd ls <strong>s3:\/\/bucket_name\/<\/strong>\n<ul>\n<li>List all objects in a bucket.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd ls s3:\/\/bucket_name\/<strong>dir\/<\/strong>\n<ul>\n<li>List all objects in a directory within a bucket.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>la<\/strong>\n<ul>\n<li>List all objects inside all buckets.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>put file.zip<\/strong> s3:\/\/bucket_name\/\n<ul>\n<li>Upload a file.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd put <strong>&#8211;multipart-chunk-size-mb=100<\/strong> file.zip s3:\/\/bucket_name\/\n<ul>\n<li>Upload a file by splitting it into 100MB chunks. The final file will appear as one.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>get <\/strong>s3:\/\/bucket_name\/file.zip\n<ul>\n<li>Download a file.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>del <\/strong>s3:\/\/bucket_name\/file.zip\n<ul>\n<li>Delete a file.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>put -r \/directory<\/strong> s3:\/\/bucket_name\/\n<ul>\n<li>Upload a directory recursively.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>mv<\/strong> s3:\/\/bucket_name\/origin\/file.zip s3:\/\/bucket_name\/destination\/file.zip\n<ul>\n<li>Move a file.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>mv<\/strong> s3:\/\/bucket_name\/origin\/ s3:\/\/bucket_name\/destination\/\n<ul>\n<li>Move a directory recursively.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>sync . -r<\/strong> s3:\/\/bucket_name\/\n<ul>\n<li>Sync a local directory with a bucket directory recursively.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>sync . &#8211;delete-removed<\/strong> s3:\/\/bucket_name\/destination\/\n<ul>\n<li>Sync a local directory with a bucket directory, deleting files that were removed from the origin.<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>signurl<\/strong> s3:\/\/bucket_name\/file.zip <strong>+600<\/strong>\n<ul>\n<li>Generate a public download URL valid for 10 minutes (600 seconds).<\/li>\n<\/ul>\n<\/li>\n<li>s3cmd <strong>signurl<\/strong> s3:\/\/bucket_name\/file.zip <strong>$(date &#8220;+%s&#8221; -d &#8220;12\/31\/2030 23:59:59&#8221;)<\/strong>\n<ul>\n<li>Generate a public download URL valid until a specified date and time.<\/li>\n<\/ul>\n<\/li>\n<li>mysqldump &#8211;all-databases<strong> | gzip |<\/strong> s3cmd put &#8211; s3:\/\/bucketName\/fileName.sql.gz\n<ul>\n<li>Pipe command output directly to an S3 bucket.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>To throttle the upload speed (to 1 Mbps) on Linux, use the following command:<\/p>\n<pre>cat mybigfile | throttle -k 1024 | s3cmd put - s3:\/\/bucket_name\/mybigfile<\/pre>\n<p>Each user&#8217;s configuration is saved in <strong>~\/.s3cfg<\/strong>.<\/p>\n<hr \/>\n<p><strong>BONUS<\/strong><\/p>\n<p>It is also possible to mount an S3 bucket in Ubuntu as a file system:<\/p>\n<pre>sudo apt install s3fs -y\r\necho <span style=\"color: #ff0000;\"><strong>ABGTKAHSICAKIAWDUVUL:JUllFbRAmydkgEz089\/UZgm\/VR1ShAZejOFiPan0<\/strong><\/span> &gt; <strong>\/etc\/passwd-s3fs<\/strong>\r\nchmod 600 <strong>\/etc\/passwd-s3fs<\/strong>\r\nmkdir \/<strong>bucket-name<\/strong>\r\ns3fs <strong>bucket-name<\/strong> \/<strong>bucket-name<\/strong><\/pre>\n<p>Note: use <strong>\/etc\/passwd-s3fs<\/strong> for system-wide access, or <strong>~\/.passwd-s3fs<\/strong> for a specific user.<\/p>\n<p>To enable debug mode, use:<\/p>\n<pre>s3fs <strong>bucket-name<\/strong> \/<strong>bucket-name<\/strong> -o dbglevel=info -f -o curldbg<\/pre>\n<p>To auto-mount the bucket on startup, add the following line to <strong>\/etc\/fstab<\/strong>:<\/p>\n<pre>s3fs#<strong>bucket-name<\/strong> \/<strong>bucket-name<\/strong> fuse _netdev,allow_other,umask=227,uid=33,gid=33,use_cache=\/root\/cache 0 0<\/pre>\n<p>To make it compatible with Linode&#8217;s S3 service, use the following command:<\/p>\n<pre>s3fs bucket-name \/bucket-name -o url=\"<strong>https:\/\/us-east-1.linodeobjects.com<\/strong>\" -o endpoint=<strong>us-east-1<\/strong> -o passwd_file=\/etc\/passwd-s3fs<\/pre>\n<p>For <strong>\/etc\/fstab<\/strong>:<\/p>\n<pre>s3fs#<strong>bucket-name<\/strong> \/<strong>bucket-name<\/strong> fuse _netdev,allow_other,use_path_request_style,url=<strong>https:\/\/us-east-1.linodeobjects.com\/<\/strong>,uid=userid,gid=groupid 0 0<\/pre>\n<p>Backup a <strong>whole directory<\/strong> (recursively), encrypt it, and push it to <strong>AWS S3<\/strong>:<\/p>\n<pre>tar -cvzf - directoryName | gpg -r \"user@domain.com\" --yes --encrypt - | aws s3 cp - s3:\/\/bucketName\/fileName.tar.gpg<\/pre>\n<hr \/>\n<p><strong>BONUS<\/strong><\/p>\n<p>Cloudflare offers S3-compatible block storage called R2. Here is how the configuration file <code>~\/.s3cfg<\/code> should look:<\/p>\n<pre>[default]\r\nhost_base = <strong>&lt;ACCOUNT_ID&gt;<\/strong>.r2.cloudflarestorage.com\r\naccess_key = <strong>&lt;ACCESS_KEY&gt;<\/strong>\r\nsecret_key = <strong>&lt;SECRET_KEY&gt;<\/strong>\r\nhost_bucket = \r\nbucket_location = auto\r\nenable_multipart = False\r\n<\/pre>\n<p>The AWS CLI tool can be used with other cloud providers, not just AWS. Below is an example configuration for connecting to Cloudflare R2, to be added to <code>~\/.aws\/credentials<\/code>:<\/p>\n<pre>[default]\r\nendpoint_url = https:\/\/<strong>&lt;ACCOUNT_ID&gt;<\/strong>.r2.cloudflarestorage.com\r\naws_access_key_id = <strong>&lt;ACCESS_KEY&gt;<\/strong>\r\naws_secret_access_key = <strong>&lt;SECRET_KEY&gt;<\/strong>\r\nregion = auto\r\noutput = json\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Installation: sudo apt update sudo apt install s3cmd -y OR sudo apt install python3-pip sudo [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,7],"tags":[],"class_list":["post-2108","post","type-post","status-publish","format-standard","hentry","category-linux","category-web"],"_links":{"self":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/2108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2108"}],"version-history":[{"count":14,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/2108\/revisions"}],"predecessor-version":[{"id":5732,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/2108\/revisions\/5732"}],"wp:attachment":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2108"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2108"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}