GlusterFS Installation and Configuration

Here are few examples how to install GlusterFS on Debian like systems. Single node, add replica node, add multiple replicas and then tweak the configuration.

Installing GlusterFS

Installing GlusterFS it’s quite simple, just type the following command in the console and it will install GlusterFS and all requirements. Also we are going to create local path /storage where we are going to save the data which will be mounted on the clients and replicated across servers when we create the replica.

apt-get install glusterfs-server
mkdir /storage

GlusterFS Single node Server (NFS Style)

If you want to have central server which will allow multiple clients to connect on it similar to NFS but the client will not have anything to setup on them, the configuration will be pulled from the server on demand. This is good if you gave multiple clients connected on the server and you will not have to connect on every client when you make some changes.

gluster volume create storage transport tcp node1:/storage force
gluster volume start storage

GlusterFS Convert to Replica

If you started with one node and now you want to add one more node and replicate the files across those two servers on node1 (The first node) type the following command:

gluster peer probe node2
gluster volume add-brick storage replica 2 node2:/storage force

Now check the volume:

gluster volume info storage

Will output:

Volume Name: storage
Type: Replicate
Volume ID: 07c46961-710e-4763-adf3-49d2c3b40eac
Status: Started
Number of Bricks: 1 x 1 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/storage
Brick2: node2:/storage

GlusterFS Distribute Replica

If you want to start from the start with two nodes from node1 type the following commands:

gluster peer probe node2
gluster volume create storage replica 3 transport tcp node1:/storage node2:/storage force
gluster volume start storage

Now check the volume:

gluster volume info storage

Will output:

Volume Name: storage
Type: Replicate
Volume ID: 07c46961-710e-4763-adf3-49d2c3b40eac
Status: Started
Number of Bricks: 1 x 1 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/storage
Brick2: node2:/storage

 

GlusterFS Distribute Replica – Four(4) nodes

This will place the files in two hosts in four volumes, reads  and writes will be distributed across this volumes. node1 and node2 will be one mirror and node3 and node4 will be another mirror.

After you install glusterfs on all 4 nodes on node1 type the folloging:

gluster peer probe node2
gluster peer probe node3
gluster peer probe node4
gluster volume create storage replica 2 transport tcp node1:/storage node2:/storage node3:/storage node4:/storage force
sudo gluster volume start storage

Now check the volume:

gluster volume info storage

Will output:

Volume Name: storage
Type: Replicate
Volume ID: 07c46961-710e-4763-adf3-49d2c3b40eac
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: node1:/storage
Brick2: node2:/storage
Brick1: node3:/storage
Brick1: node4:/storage

Connecting GlusterFS Clients

If you need only client software installed on the host you can use the following command:

apt-get install glusterfs-client

Now we can test and see if we can connect to the GlusterFS server node1:

mount -t glusterfs node1:/storage /path/on/client/machine

If everything is OK you can add it in /etc/fstab

node1:/storage /path/on/client/machine glusterfs defaults,_netdev 0 0
mount -a

GlusterFS Performance tweaks

Here are some performance tweaks that will be passed to the clients automatically:

gluster volume set storage nfs.enable-ino32 on
gluster volume set storage nfs.addr-namelookup off
gluster volume set storage nfs.disable on
gluster volume set storage performance.cache-max-file-size 2MB
gluster volume set storage performance.cache-refresh-timeout 4
gluster volume set storage performance.cache-size 256MB
gluster volume set storage performance.write-behind-window-size 4MB
gluster volume set storage performance.io-thread-count 32

GlusterFS Quota settings

The nodes will be large and planed for a lot of data, but eventually you will start setting up quota. It can be done and the whole volume and on specific directories:

gluster volume quota storage enable
gluster volume quota storage limit-usage / 50GB
gluster volume quota storage limit-usage /path/to/some/directory 20G

Tweak it for faster timeout and list quotas and remove the limits if necessary

gluster volume set storage features.quota-timeout 5
gluster volume quota storage list 
gluster volume quota storage remove /path/to/some/directory

I hope this will help you get started with GlusterFS

GlusterFS: https://www.gluster.org/

 

Nikola Stojanoski

System Administrator and Developer. Giving back to the community by blogging about my problems, solutions and practical howto's.