{"id":4830,"date":"2025-05-30T20:50:38","date_gmt":"2025-05-31T00:50:38","guid":{"rendered":"https:\/\/dft.wiki\/?p=4830"},"modified":"2026-06-08T09:48:34","modified_gmt":"2026-06-08T13:48:34","slug":"glusterfs-cheat-sheet","status":"publish","type":"post","link":"https:\/\/dft.wiki\/?p=4830","title":{"rendered":"GlusterFS Cheat Sheet"},"content":{"rendered":"<p><strong>GlusterFS<\/strong> is an open-source distributed storage platform developed by Red Hat, designed for scale-out storage in both public and private cloud environments.<br \/>\nWhile modern applications are increasingly scalable thanks to container orchestrators like Kubernetes, persistent and shared storage often struggles to scale in the same way.<br \/>\n<img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4834\" src=\"https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-19-16.png\" alt=\"\" width=\"289\" height=\"237\" \/><br \/>\nBy using a distributed file system like GlusterFS, you can achieve <strong>resilience<\/strong> through data replication across multiple servers. Additionally, <strong>performance<\/strong> can be improved by <strong>distributing and striping<\/strong> data, enabling simultaneous read and write operations across many servers.<\/p>\n<ul>\n<li><strong>Replicated<\/strong>\n<ul>\n<li>The same file is copied on each server for redundancy.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Distributed<\/strong> (default)\n<ul>\n<li>Files are spread over multiple servers for best throughput.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Dispersed<\/strong>\n<ul>\n<li>An encoded fragment of the file is stored on each brick, but only a subset of fragments is needed to recover it.<\/li>\n<li>Usable_Size = Brick_Size * (Number_of_Bricks &#8211; Redundancy)<\/li>\n<\/ul>\n<\/li>\n<li><strong>Distributed Replicated<\/strong> or <strong>Distributed Dispersed<\/strong>\n<ul>\n<li>A combination of the previous methods.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>For reference, here is how they compare to RAID.<\/p>\n<ul>\n<li><strong>RAID 0<\/strong>\n<ul>\n<li>Striping with no redundancy.<\/li>\n<li>Similar to <strong>GlusterFS Distributed<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<li><strong>RAID 1<\/strong>\n<ul>\n<li>Mirroring with full copies.<\/li>\n<li>Similar to <strong>GlusterFS Replicated<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<li><strong>RAID 5<\/strong>\n<ul>\n<li>Striping with single parity.<\/li>\n<li>Similar to <strong>GlusterFS Dispersed with Redundancy of 1<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<li><strong>RAID 6<\/strong>\n<ul>\n<li>Striping with double parity.<\/li>\n<li>Similar to <strong>GlusterFS Dispersed with Redundancy of 2<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<li><strong>RAID 10<\/strong>\n<ul>\n<li>Mirroring combined with striping.<\/li>\n<li>Similar to <strong>GlusterFS Distributed Replicated<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr \/>\n<p><strong>INSTALLING GLUSTERFS SERVER<\/strong><\/p>\n<p>Run the following commands on all nodes in the cluster.<\/p>\n<pre>apt update &amp;&amp; apt upgrade -y &amp;&amp; apt install software-properties-common -y\r\nadd-apt-repository ppa:gluster\/glusterfs-11\r\napt install glusterfs-server -y\r\nsystemctl enable --now glusterd\r\nnano \/etc\/hosts<\/pre>\n<p>Append the following lines, adjusting the IPs and number of servers as needed.<\/p>\n<pre>10.1.1.11 g1\r\n10.1.1.12 g2\r\n10.1.1.13 g3<\/pre>\n<p>On one of the servers, run the following commands to probe its peers and add them to the cluster.<\/p>\n<pre>gluster peer probe g2\r\ngluster peer probe g3\r\ngluster peer status<\/pre>\n<p>On all servers, create the directories that will be used by the volumes.<\/p>\n<pre>mkdir -p \/glusterfs\/brick1\r\nmkdir -p \/glusterfs\/brick2<\/pre>\n<p>The first volume stores data in a replicated fashion. In this example, three replicas of the same data are always maintained (200% overhead).<\/p>\n<pre>gluster volume create gv1 replica 3 transport tcp g1:\/glusterfs\/brick1 g2:\/glusterfs\/brick1 g3:\/glusterfs\/brick1 force\r\ngluster volume start gv1<\/pre>\n<p>The second volume is distributed but not redundant (0% overhead). This maximizes usable storage and throughput at the cost of data integrity, making it well suited for ephemeral data such as logs.<\/p>\n<pre>gluster volume create gv2 transport tcp g1:\/glusterfs\/brick2 g2:\/glusterfs\/brick2 g3:\/glusterfs\/brick2 force\r\ngluster volume start gv2\r\ngluster volume info<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4835\" src=\"https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-20-25.png\" alt=\"\" width=\"386\" height=\"541\" srcset=\"https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-20-25.png 386w, https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-20-25-214x300.png 214w\" sizes=\"auto, (max-width: 386px) 100vw, 386px\" \/><\/p>\n<pre>gluster volume status<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4836\" src=\"https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-21-54.png\" alt=\"\" width=\"630\" height=\"395\" srcset=\"https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-21-54.png 630w, https:\/\/dft.wiki\/wp-content\/uploads\/sites\/15\/2025\/05\/Screenshot_2025-05-30_21-21-54-300x188.png 300w\" sizes=\"auto, (max-width: 630px) 100vw, 630px\" \/><\/p>\n<p>For a Distributed Replicated volume, the number of bricks must be a multiple of the replica count. Example: 2 replicas and 4 bricks.<\/p>\n<pre>gluster volume create gv3 replica <strong>2<\/strong> g1:\/<strong>brick<\/strong> g2:\/<strong>brick<\/strong> g3:\/<strong>brick<\/strong> g4:\/<strong>brick<\/strong><\/pre>\n<p>Then open the required ports on the firewall and enable it.<\/p>\n<pre># Control Ports\r\nufw allow 24007\r\nufw allow 24008\r\n# Data Ports (one for each brick, starting from 49152)\r\nufw allow 49152\r\nufw allow 49153\r\n# Enable UFW\r\nufw enable<\/pre>\n<hr \/>\n<p><strong>INSTALLING GLUSTERFS CLIENT<\/strong><\/p>\n<pre>apt update &amp;&amp; apt upgrade -y &amp;&amp; apt install software-properties-common -y\r\nadd-apt-repository ppa:gluster\/glusterfs-11\r\napt install glusterfs-client -y<\/pre>\n<p>Create the mount points for the volumes and mount them.<\/p>\n<pre>mkdir -p \/mnt\/gv1\r\nmkdir -p \/mnt\/gv2\r\nmount -t glusterfs g1:\/gv1 \/mnt\/gv1\r\nmount -t glusterfs g1:\/gv2 \/mnt\/gv2<\/pre>\n<p>If successful, unmount them and configure automatic mounting on boot.<\/p>\n<pre>umount \/mnt\/gv1\r\numount \/mnt\/gv2\r\nnano \/etc\/fstab<\/pre>\n<p>Add the following lines at the end.<\/p>\n<pre>g1:\/gv1 \/mnt\/gv1 glusterfs defaults,_netdev,backupvolfile-server=g3 0 0\r\ng2:\/gv2 \/mnt\/gv2 glusterfs defaults,_netdev,backupvolfile-server=g3 0 0<\/pre>\n<p><strong>Note:<\/strong> For volume 1, the client connects via server 1 by default, with server 3 as a fallback. For volume 2, server 2 is used by default, also with server 3 as a fallback. Once the initial connection is established, the client becomes aware of all servers and communicates with them directly and simultaneously.<\/p>\n<p>Apply the fstab changes without rebooting.<\/p>\n<pre>mount -a<\/pre>\n<hr \/>\n<p><strong>MANAGING VOLUMES<\/strong><\/p>\n<pre>gluster volume list\r\ngluster volume info [&lt;VOLUME_NAME&gt;]\r\ngluster volume start &lt;VOLUME_NAME&gt;\r\ngluster volume stop &lt;VOLUME_NAME&gt;\r\ngluster volume stop &lt;VOLUME_NAME&gt; --mode=force\r\ngluster volume delete &lt;VOLUME_NAME&gt;<\/pre>\n<p>Scale Out<\/p>\n<pre>gluster volume add-brick &lt;VOLUME_NAME&gt; &lt;HOST&gt;:\/brick_path\r\ngluster volume add-brick &lt;VOLUME_NAME&gt; replica &lt;count&gt; &lt;HOSTS:\/brick_paths...&gt;<\/pre>\n<p>Scale In<\/p>\n<pre>gluster volume remove-brick &lt;VOLUME_NAME&gt; &lt;HOST&gt;:\/brick_path start\r\n# Wait for data evacuation\r\ngluster volume remove-brick &lt;VOLUME_NAME&gt; &lt;HOST&gt;:\/brick_path commit<\/pre>\n<p>Rebalance a Volume<\/p>\n<pre>gluster volume rebalance &lt;VOLUME_NAME&gt; start\r\ngluster volume rebalance &lt;VOLUME_NAME&gt; status<\/pre>\n<p>Volume Options<\/p>\n<pre>gluster volume set &lt;VOLUME_NAME&gt; &lt;OPTION&gt; &lt;VALUE&gt;\r\ngluster volume set gv0 performance.cache-size 256MB<\/pre>\n<p>Quotas<\/p>\n<pre>gluster volume quota &lt;VOLUME_NAME&gt; list\r\ngluster volume quota &lt;VOLUME_NAME&gt; enable\r\ngluster volume quota &lt;VOLUME_NAME&gt; limit-usage \/ &lt;SIZE&gt;\r\ngluster volume quota &lt;VOLUME_NAME&gt; limit-usage \/&lt;DIR_PATH&gt; &lt;SIZE&gt;\r\ngluster volume quota &lt;VOLUME_NAME&gt; remove \/&lt;DIR_PATH&gt;\r\ngluster volume quota &lt;VOLUME_NAME&gt; disable<\/pre>\n<p>Access Control<\/p>\n<pre>gluster volume set &lt;VOLUME_NAME&gt; auth.allow &lt;IP_1,IP_2,...&gt;\r\ngluster volume set &lt;VOLUME_NAME&gt; auth.reject &lt;IP_1,IP_2,...&gt;<\/pre>\n<hr \/>\n<p><strong>BONUS<\/strong><\/p>\n<p>Create persistent storage in Kubernetes.<\/p>\n<p>Service<\/p>\n<pre>apiVersion: v1\r\nkind: Service\r\nmetadata:\r\n  name: glusterfs-cluster\r\nspec:\r\n  ports:\r\n  - port: 24007\r\n    targetPort: 24007\r\n    protocol: TCP\r\n    name: gluster-management<\/pre>\n<p>Endpoint<\/p>\n<pre>apiVersion: v1\r\nkind: Endpoints\r\nmetadata:\r\n  name: glusterfs-cluster\r\nsubsets:\r\n  - addresses:\r\n      - ip: 10.1.1.11\r\n      - ip: 10.1.1.12\r\n      - ip: 10.1.1.13\r\n    ports:\r\n      - port: 24007\r\n        protocol: TCP<\/pre>\n<p>Persistent Volume<\/p>\n<pre>apiVersion: v1\r\nkind: PersistentVolume\r\nmetadata:\r\n  name: gluster-pv\r\nspec:\r\n  capacity:\r\n    storage: 10Gi\r\n  accessModes:\r\n    - ReadWriteMany\r\n  persistentVolumeReclaimPolicy: Retain\r\n  glusterfs:\r\n    endpoints: glusterfs-cluster\r\n    path: gv0\r\n    readOnly: false<\/pre>\n<p>Persistent Volume Claim<\/p>\n<pre>apiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n  name: gluster-pvc\r\nspec:\r\n  accessModes:\r\n    - ReadWriteMany\r\n  resources:\r\n    requests:\r\n      storage: 10Gi\r\n  volumeName: gluster-pv<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>GlusterFS is an open-source distributed storage platform developed by Red Hat, designed for scale-out storage [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,1],"tags":[],"class_list":["post-4830","post","type-post","status-publish","format-standard","hentry","category-linux","category-ccna"],"_links":{"self":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/4830","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4830"}],"version-history":[{"count":7,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/4830\/revisions"}],"predecessor-version":[{"id":5609,"href":"https:\/\/dft.wiki\/index.php?rest_route=\/wp\/v2\/posts\/4830\/revisions\/5609"}],"wp:attachment":[{"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4830"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4830"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dft.wiki\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4830"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}