by: Linux/Unix Web

The original purpose of this post was to showcase and walk through the steps to deploy and operate MinIO, which was the most widely deployed S3-compatible object storage solution prior to its rugpull for profit. This is a long story I will not cover here, but the Community Edition is currently out of support.

Even though it is not recommended to use MinIO in production or to leave it directly exposed to the internet, I will keep the original instructions at the end of this post and highlight the most popular alternatives:

RUSTFS

I want to start with RustFS because it is the one that impressed me the most with how easy it is to deploy and use [Link]. The language it is written in should be self-explanatory.

It integrates well with TrueNAS CE for quick deployment.

For deploying on Linux systems:

curl -O https://rustfs.com/install_rustfs.sh && sudo bash install_rustfs.sh

For use with TLS certificates such as Let’s Encrypt:

sudo mkdir -p /opt/tls
echo 'RUSTFS_TLS_PATH="/opt/tls"' | sudo tee -a /etc/default/rustfs
sudo sed 's/9000/443/g' /etc/default/rustfs -i

Place the certificate and key files in that directory with the correct names and RustFS will load them at startup:

cat /etc/letsencrypt/live/example.com/privkey.pem   | sudo tee /opt/tls/rustfs_key.pem
cat /etc/letsencrypt/live/example.com/fullchain.pem | sudo tee /opt/tls/rustfs_cert.pem

Restart the service:

systemctl restart rustfs

Alternatively, deploy RustFS as a Docker container:

docker run -d --name rustfs -e RUSTFS_TLS_PATH="/opt/tls/" -v /opt/tls:/opt/tls -p 9000:443 -p 9001:9001 -v /data:/data rustfs/rustfs:latest

In all cases, make sure the Console Dashboard is not directly exposed to the internet without proper security layers, such as a VPN or an IP allowlist.

GARAGE

(in-progress)

SEAWEEDFS

(in-progress)

MINIO (unmaintained)

MinIO was an open source [Link] high-performance object storage API written in Go, compatible with Amazon S3 cloud storage. It can handle unstructured data (photos, videos, log files, backups, container images, etc.) with objects up to 5 TB in size.

Its enterprise-class features support striping files across multiple disks and reliably recovering data even when multiple drives fail. Based on its parity/hash logic, it also heals corrupted files on the fly to ensure data integrity.

MinIO also features server-side encryption, continuous site replication, bucket versioning, deletion protection, event notifications, lifecycle management, access rules, clustering, and integration with public cloud services.

DEPLOYMENT METHODS

  • From a binary:
    • Download the binary for your CPU architecture and run it directly with no installation required, or install it via your package manager.
  • From a container:
    • A single command spins up a new instance. Great for evaluation and development.
  • From source code:
    • Using Go’s native tooling, you can pull and compile the binary yourself. This is the most secure option.

BINARY EXECUTION

  • Linux 
    • export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=strongpassword
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /PATH/data --console-address ":9001"
  • ARM (Raspberry Pi) 
    • export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=strongpassword
wget https://dl.min.io/server/minio/release/linux-arm64/minio
chmod +x minio
./minio server /PATH/data --console-address ":9001"
  • macOS 
    • export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=strongpassword
wget https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio
./minio server /PATH/data --console-address ":9001"
  • Windows 
    • PS> Invoke-WebRequest -Uri "https://dl.min.io/server/minio/release/windows-amd64/minio.exe" -OutFile "C:\minio.exe"
PS> setx MINIO_ROOT_USER admin
PS> setx MINIO_ROOT_PASSWORD strongpassword
PS> C:\minio.exe server D:\Data --console-address ":9001"

PACKAGE MANAGER INSTALLATION

  • Linux
    • For Debian-based distributions: 
      • wget https://dl.min.io/server/minio/release/linux-amd64/minio_20220620231345.0.0_amd64.deb
sudo dpkg -i minio_20220620231345.0.0_amd64.deb
    • For RHEL-based distributions: 
      • wget https://dl.min.io/server/minio/release/linux-amd64/minio-20220620231345.0.0.x86_64.rpm
sudo rpm -i minio-20220620231345.0.0.x86_64.rpm
  • ARM (Raspberry Pi)
    • For Debian-based distributions: 
      • wget https://dl.min.io/server/minio/release/linux-arm64/minio_20220620231345.0.0_arm64.deb
sudo dpkg -i minio_20220620231345.0.0_amd64.deb
    • For RHEL-based distributions: 
      • wget https://dl.min.io/server/minio/release/linux-arm64/minio-20220620231345.0.0.aarch64.rpm
sudo rpm -i minio-20220620231345.0.0.x86_64.rpm
  • macOS 
    • brew install minio/stable/minio
minio server start

DOCKER DEPLOYMENT

  • Using Docker: 
    • sudo docker run -p 9000:9000 -p 9001:9001 -v /PATH/data:/data quay.io/minio/minio server /PATH/data --console-address ":9001"
  • Using Podman: 
    • podman run -p 9000:9000 -p 9001:9001 -v /PATH/data:/data quay.io/minio/minio server /PATH/data --console-address ":9001"

COMPILING FROM SOURCE

  • Using Go: 
    • GO111MODULE=on go install github.com/minio/minio@latest
  • Manually: 
    • git clone https://github.com/minio/minio.git
cd minio/
make
make install

FIREWALL SETTINGS

  • UFW 
    • sudo ufw allow 9000:9001/tcp
  • firewall-cmd 
    • sudo firewall-cmd --zone=public --add-port=9000-9001/tcp --permanent
sudo firewall-cmd --reload

BONUS

  • Securing 
    • sudo useradd -r minioUser -s /sbin/nologin
sudo mkdir -p /PATH/data
sudo chown minioUser:minioUser /PATH/data
sudo chmod 770 /PATH/data
  • Configuration File 
    • sudo nano /etc/default/minio
    • MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=strongpassword 
MINIO_VOLUMES="/PATH/data"
MINIO_OPTS="--certs-dir /home/minioUser/.minio/certs --console-address :9001"
    • sudo chown minioUser:minioUser /etc/default/minio
sudo chmod 770 /etc/default/minio
  • Running as a Service 
    • sudo nano /etc/systemd/system/minio.service
    • [Unit]
Description=MinIO
Wants=network-online.target
After=network-online.target

[Service]
WorkingDirectory=/usr/local/
User=minioUser
Group=minioUser
ProtectProc=invisible

EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
LimitNOFILE=1048576
TasksMax=infinity
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
    • sudo systemctl daemon-reload
sudo systemctl enable minio
sudo systemctl start minio
sudo systemctl status minio
  • MinIO Client
    • Download 
      • wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help
sudo mv mc /usr/local/bin
mc --help
    • Configuration 
      • ./mc alias set <ALIAS> http://<YOUR-S3-ENDPOINT>:9000 <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY>
./mc ls <ALIAS>
    • Basic commands (<ALIAS> = s3)
      • mc ls s3
        • List buckets.
      • mc mb s3/new-bucket
        • Create a new bucket.
      • mc rm s3/new-bucket
        • Delete a bucket.
      • mc ls s3/bucket
        • List the contents of a bucket.
      • mc cp localFile s3/bucket/remoteFile
        • Copy a file from local to remote.
      • mc cp s3/bucket/remoteFile localFile
        • Copy a file from remote to local.
      • mc cp s3/backup/fileName s3/backup/copiedFile
        • Copy a file within the same bucket (also works between buckets).
      • mc mv s3/backup/fileName s3/backup/renamedFile
        • Rename or move a file within the same bucket (also works between buckets).
      • mc rm s3/backup/fileToRemove
        • Delete a file (object).
      • mc find s3 –name “*.zip”
        • Search all buckets for files by name (also supports regex, max depth, age, and more).
      • mc find s3/bucket –name “*.zip” –exec “mc cp {} .”
        • Run an external command for each matching object, such as copying it to the current directory.
      • mc mirror backup/ s3/backup
        • Recursively sync a local directory to a remote location (also works between remote locations).
      • mc update
        • Update the client to the latest version.
    • Additional commands
      • cat
      • head
      • pipe
      • sql
      • stat
      • tree
      • du
      • retention
      • legalhold
      • share
      • version
      • ilm
      • encrypt
      • event
      • watch
      • undo
      • anonymous
      • tag
      • diff
      • replicate
    • Administrative commands
      • mc admin config get s3 site
        • Get configuration information.
      • mc admin config set s3 site name=mys3
        • Set configuration information.
      • mc admin service restart s3
        • Restart the service.
    • Upgrading 
      • mc admin update s3
    • Backups 
      • tar zcf - /PATH | mc pipe -q <ALIAS>/<BUCKET>/fileName.tar.gz
mysqldump --no-tablespaces --lock-tables=false dbName | gzip -c | mc pipe -q <ALIAS>/<BUCKET>/fileName.tar.gz
    • Versioning and Lifecycle 
      • mc mb --with-lock <ALIAS>/<BUCKET>
mc version enable <ALIAS>/<BUCKET>
mc ilm rule add --noncurrent-expire-days 30 <ALIAS>/<BUCKET>
mc ilm rule add --expire-days 90 --noncurrent-expire-days 30 <ALIAS>/<BUCKET>
mc ls --versions <ALIAS>/<BUCKET>
  • Encrypting
    • I recommend placing a web server such as Apache, NGINX, or Varnish between the client and the MinIO server to simplify certificate management, virtual hosting, load balancing, and caching.
    • That said, MinIO has built-in support for encrypted communication using either a self-signed certificate or a public certificate from Let’s Encrypt via Certbot: 
      • sudo apt install snapd
sudo snap install core 
sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot certonly --standalone --agree-tos --no-eff-email --staple-ocsp --preferred-challenges http -m [email protected] -d minio.domain.com
sudo cp /etc/letsencrypt/live/minio.domain.com/privkey.pem /home/minioUser/.minio/certs/private.key
sudo cp /etc/letsencrypt/live/minio.domain.com/fullchain.pem /home/minioUser/.minio/certs/public.crt
sudo ufw allow 80
sudo ufw allow 443
      • sudo nano /etc/default/minio
      • MINIO_SERVER_URL="https://minio.domain.com:9000"
      • sudo systemctl restart minio
  • Optional environment variables 
    • export MINIO_DOMAIN=domain.com
export MINIO_SITE_NAME=mysite
export MINIO_SITE_REGION=global
export MINIO_SITE_COMMENT="My Block Storage"
export MINIO_API_REQUESTS_MAX=1600
export MINIO_API_REQUESTS_DEADLINE=1m
export MINIO_API_CORS_ALLOW_ORIGIN="https://domain-a.com,https://domain-b.com"
export MINIO_API_REMOTE_TRANSPORT_DEADLINE=2h
export MINIO_BROWSER=off
  • Multiple resources
    • Maps resources based on the host address: 
      • minio server https://minio-{1...4}.example.net/mnt/disk-{1...4} --console-address ":9001"
  • Directory Tree
    • Certificates are stored by default in the home directory of the user running the binary:
      • /home/minioUser/.minio/ 
        • └── certs
    └── CAs
    • All instance configuration is stored in a hidden directory inside the data volume:
      • /PATH/data/.minio.sys/ 
        • ├── buckets
├── config
│   ├── config.json
│   │   └── xl.meta
│   └── iam
│       ├── format.json
│       │   └── xl.meta
│       ├── policydb
│       │   └── users
│       │       └── username.json
│       │           └── xl.meta
│       ├── sts
│       │   ├── 2G67ZRC44SCKLVLLZVGT
│       │   │   └── identity.json
│       │   │       └── xl.meta
│       │   ├── 3KEUK011NRQIOOS0AK0R
│       │   │   └── identity.json
│       │   │       └── xl.meta
│       │   └── AAQ83Z1Z279SWO7VMH5T
│       │       └── identity.json
│       │           └── xl.meta
│       └── users
│           └── username
│               └── identity.json
│                   └── xl.meta
├── format.json
├── ilm
│   └── deletion-journal.bin
├── multipart
├── tmp
└── tmp-old

MinIO CE underwent a major overhaul that removed several administrative features from its Console. Here are some workarounds you may need.

Deploying from source:

sudo snap install go --classic
go install -v github.com/minio/minio@latest
sudo mkdir /data
sudo chown minio:minio /data
sudo chown -R minio:minio /data
sudo nano /etc/systemd/system/minio.service

To run it as a service:

[Unit]
Description=MinIO
Wants=network-online.target
After=network-online.target

[Service]
User=minio
Group=minio
ProtectProc=invisible

Environment="MINIO_ROOT_USER=admin_user"
Environment="MINIO_ROOT_PASSWORD=strong_password"
Environment="MINIO_BROWSER_REDIRECT_URL=https://minio-console.simnet.cloud"
ExecStart=/home/minio/go/bin/minio server /data --console-address :9001
Restart=always
LimitNOFILE=1048576
TasksMax=infinity
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

Load and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now minio
journalctl -u minio -f
  • If using NGINX as a reverse proxy:
    • Remove or increase the upload size limit.
    • The new file browser in the UI requires WebSocket.
client_max_body_size 0;
location / {
  proxy_pass http://127.0.0.1:9001;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_connect_timeout 300;
  proxy_send_timeout 300;
  proxy_read_timeout 300;
  chunked_transfer_encoding on;
  # WebSocket
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
}
  • Create a bucket with versioning and locking via the CLI:
mc mb --with-lock minio/acs
mc retention set --default GOVERNANCE "30d" minio/acs

READ MORE

Configure a Reverse Proxy with HTTP for MinIO [Link].

Configure a Reverse Proxy with Cache for MinIO [Link].

Check out the backup tools Duplicati (open source) [Link], Duplicity [Link], and Duplicacy [Link], all of which are S3 compatible, among many other features.