How to setup a network shared folder using GlusterFS on Ubuntu Servers with backup server array and auto mount

How to setup a network shared folder using GlusterFS on Ubuntu Servers with backup server array and auto mount

Introduction

GlusterFS, NFS and Samba are the most popular 3 ways to setup a network shared folder on Linux.

In this tutorial, we will learn how to setup a network shared folder using GlusterFS on Ubuntu Servers.

For demonstration purposes, we will setup the shared folder on two server machines, server1.example.com and server2.example.com, and mount that shared folder on one client machine, client1.example.com.

Let’s say the servers’ IP’s are 10.128.0.1 and 10.128.0.2, and the client’s IP is 10.128.0.3.

Prepare the enviroment

Configure DNS resolution

In order for our different components to be able to communicate with each other easily, it is best to set up some kind of hostname resolution between each computer.

The easiest way to do this is editing the hosts file on each computer.

Open this file with root privileges on your first computer:

$ sudo nano /etc/hosts

You should see something that looks like this:

127.0.0.1       localhost client1

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Below the local host definition, you should add each VPS’s IP address followed by the long and short names you wish to use to reference it.

It should look something like this when you are finished:

server1/server2/client1

127.0.0.1       localhost hostname

# add our machines here
10.128.0.1 server1.example.com server1
10.128.0.2 server2.example.com server2
10.128.0.3 client1.example.com client1
# end add our machines

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

The server1.example.com and server1 portions of the lines can be changed to whatever name you would like to use to access each computer. We will be using these settings for this guide.

When you are finished, copy the lines you added and add them to the /etc/hosts files on your other computer. Each /etc/hosts file should contain the lines that link your IPs to the names you’ve selected.

Save and close each file when you are finished.

Set Up Software Sources

Although Ubuntu 12.04 contains GlusterFS packages, they are fairly out-of-date, so we will be using the latest stable version as of the time of this writing (version 3.4) from the GlusterFS project.

We will be setting up the software sources on all of the computers that will function as nodes within our cluster, as well as on the client computer.

We will actually be adding a PPA (personal package archive) that the project recommends for Ubuntu users. This will allow us to manage our packages with the same tools as other system software.

First, we need to install the python-software-properties package, which will allow us to manage PPAs easily with apt:

server1/server2/client1

$ sudo apt-get update
$ sudo apt-get install python-software-properties

Once the PPA tools are installed, we can add the PPA for the GlusterFS packages by typing:

server1/server2/client1

$ sudo add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.4

With the PPA added, we need to refresh our local package database so that our system knows about the new packages available from the PPA:

server1/server2/client1

$ sudo apt-get update

Repeat these steps on all of the VPS instances that you are using for this guide.

Install Server Components

On our cluster member machines (server1 and server2), we can install the GlusterFS server package by typing:

server1/server2

$ sudo apt-get install glusterfs-server

Once this is installed on both nodes, we can begin to set up our storage volume.

On one of the hosts, we need to peer with the second host. It doesn’t matter which server you use, but we will be preforming these commands from our server1 server for simplicity:

server1

$ sudo gluster peer probe server2.example.com

Console should output:

peer probe: success

This means that the peering was successful. We can check that the nodes are communicating at any time by typing:

server1

$ sudo gluster peer status

Console should output:

Number of Peers: 1

Hostname: server2.example.com
Port: 24007
Uuid: 7bcba506-3a7a-4c5e-94fa-1aaf83f5729b
State: Peer in Cluster (Connected)

At this point, our two servers are communicating and they can set up storage volumes together.

Create a Storage Volume

Now that we have our pool of servers available, we can make our first volume.

This step needs to be run only on either one of the two servers. In this guide we will be running from server1.

Because we are interested in redundancy, we will set up a volume that has replica functionality. This will allow us to keep multiple copies of our data, saving us from a single point-of-failure.

Since we want one copy of data on each of our servers, we will set the replica option to “2”, which is the number of servers we have. The general syntax we will be using to create the volume is this:

$ sudo gluster volume create volume_name replica num_of_servers transport tcp domain1.com:/path/to/data/directory domain2.com:/path/to/data/directory force

The exact command we will run is this:

server1

$ sudo gluster volume create volume1 replica 2 transport tcp server1.example.com:/gluster-storage server2.example.com:/gluster-storage force

The console would output like this:

volume create: volume1: success: please start the volume to access data

This will create a volume called volume1. It will store the data from this volume in directories on each host at /gluster-storage. If this directory does not exist, it will be created.

At this point, our volume is created, but inactive. We can start the volume and make it available for use by typing:

server1

$ sudo gluster volume start volume1

Console should output:

volume start: volume1: success

Our volume should be online currently.

Install and Configure the Client Components

Now that we have our volume configured, it is available for use by our client machine.

Before we begin though, we need to actually install the relevant packages from the PPA we set up earlier.

On your client machine (client1 in this example), type:

client1

$ sudo apt-get install glusterfs-client

This will install the client application, and also install the necessary fuse filesystem tools necessary to provide filesystem functionality outside of the kernel.

We are going to mount our remote storage volume on our client computer. In order to do that, we need to create a mount point. Traditionally, this is in the /mnt directory, but anywhere convenient can be used.

We will create a directory at /storage-pool:

client1

$ sudo mkdir /storage-pool

With that step out of the way, we can mount the remote volume. To do this, we just need to use the following syntax:

$ sudo mount -t glusterfs domain1.com:volume_name path_to_mount_point

Notice that we are using the volume name in the mount command. GlusterFS abstracts the actual storage directories on each host. We are not looking to mount the /gluster-storage directory, but the volume1 volume.

Also notice that we only have to specify one member of the storage cluster.

The actual command that we are going to run is this:

client1

$ sudo mount -t glusterfs server1.example.com:/volume1 /storage-pool

This should mount our volume. If we use the df command, you will see that we have our GlusterFS mounted at the correct location.

Testing the Redundancy Features

Now that we have set up our client to use our pool of storage, let’s test the functionality.

On our client machine (client1), we can type this to add some files into our storage-pool directory:

client1

$ cd /storage-pool
$ sudo touch file{1..20}

This will create 20 files in our storage pool.

If we look at our /gluster-storage directories on each storage host, we will see that all of these files are present on each system:

server1/server2

# on server1.example.com and server2.example.com
$ cd /gluster-storage
$ ls

Console should output:

file1  file10  file11  file12  file13  file14  file15  file16  file17  file18  file19  file2  file20  file3  file4  file5  file6  file7  file8  file9

As you can see, this has written the data from our client to both of our nodes.

If there is ever a point where one of the nodes in your storage cluster is down and changes are made to the filesystem. Doing a read operation on the client mount point after the node comes back online should alert it to get any missing files:

client1

$ ls /storage-pool

Set Up Backup Server(s) for the Client

Normally, once client1 has connected to server1, server1 will send all the nodes’ information to client1, so that client1 can connect to any node in the pool to get the data afterwards.

However, in our set up so far, if server1 is not available before client1 first connects to server1, client1 will not know about server2 and therefore can’t connect to our gluster volume.

To enable client1 to connect to server2 when server1 is not available, we can use the option backupvolfile-server as following

client1

$ sudo mount -t glusterfs server1.example.com:/volume1 /storage-pool -o backupvolfile-server=server2.example.com

If our gluster pool has more then one backup server, we can list all the server using the backupvolfile-servers as following (notice the plural s at the end of the param)

client1

$ sudo mount -t glusterfs server1.example.com:/volume1 /storage-pool -o backupvolfile-servers=server2.example.com:server3.example.com:server4.example.com

Set Up Auto Mounting on the Client

In theory adding the following line to the client’s fstab file should make the client mount the GlusterFS share at boot:

client1

server1.example.com:/volume1 /storage-pool glusterfs defaults,_netdev 0 0

Normally this should work since the _netdev param should force the filesystem to wait for a network connection.

If this didn’t work for you because the GlusterFS client wasn’t running when the fstab file was processed, try opening root’s crontab file and add a command to mount the share at reboot. This command opens the crontab file:

client1

$ sudo crontab -u root -e

Add this line, and press control-o and return to save changes, and control-x to quit from nano:

client1

@reboot sleep 10;mount -t glusterfs server1.example.com:/volume1 /storage-pool -o backupvolfile-server=server2.example.com

This will execute two commands when the server boots up: the first is just a 10 second delay to allow the GlusterFS daemon to boot, and the second command mounts the volume.

You may need to make your client wait longer before running mount. If your client doesn’t mount the volume when it boots, try using ‘sleep 15’ instead.  This isn’t an ideal way to fix this problem, but it’s ok for most uses.

Another appropriate way to setup auto mounting is that instead of mounting the GlusterFS share manually on the client, you add the mount command to /etc/rc.local file. We will not add it to /etc/fstab as rc.local is always executed after the network is up which is required for a network file system.

Open /etc/rc.local

client1

$ nano /etc/rc.local

Append the following line:

client1

[...]
/usr/sbin/mount.glusterfs server1.example.com:/volume1 /storage-pool

To test if your modified /etc/rc.local is working, reboot the client:

$ reboot

After the reboot, you should find the share in the outputs of…

$ df -h

… and…

$ mount

Restrict Access to the Volume

Now that we have verified that our storage pool can be mounted and replicate data to both of the machines in the cluster, we should lock down our pool.

Currently, any computer can connect to our storage volume without any restrictions. We can change this by setting an option on our volume.

On one of your storage nodes, type:

server1

$ sudo gluster volume set volume1 auth.allow gluster_client_IP_addr

You will have to substitute the IP address of your cluster client (client1) in this command. Currently, at least with /etc/hosts configuration, domain name restrictions do not work correctly. If you set a restriction this way, it will block all traffic. You must use IP addresses instead.

If you need to remove the restriction at any point, you can type:

server1

$ sudo gluster volume set volume1 auth.allow *

This will allow connections from any machine again. This is insecure, but may be useful for debugging issues.

If you have multiple clients, you can specify their IP addresses at the same time, separated by commas:

server1

$ sudo gluster volume set volume1 auth.allow gluster_client1_ip,gluster_client2_ip

Getting Info with GlusterFS Commands

When you begin changing some of the settings for your GlusterFS storage, you might get confused about what options you have available, which volumes are live, and which nodes are associated with each volume.

There are a number of different commands that are available on your nodes to retrieve this data and interact with your storage pool.

If you want information about each of your volumes, type:

server1/server2

$ sudo gluster volume info

Console output:

Volume Name: volume1
Type: Replicate
Volume ID: 3634df4a-90cd-4ef8-9179-3bfa43cca867
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.example.com:/gluster-storage
Brick2: server2.example.com:/gluster-storage
Options Reconfigured:
auth.allow: 111.111.1.11

Similarly, to get information about the peers that this node is connected to, you can type:

server1/server2

$ sudo gluster peer status

Console output:

Number of Peers: 1

Hostname: server1.example.com
Port: 24007
Uuid: 6f30f38e-b47d-4df1-b106-f33dfd18b265
State: Peer in Cluster (Connected)

If you want detailed information about how each node is performing, you can profile a volume by typing:

server1/server2

$ sudo gluster volume profile volume_name start

When this command is complete, you can obtain the information that was gathered by typing:

server1/server2

$ sudo gluster volume profile volume_name info

Console output:

Brick: server2.example.com:/gluster-storage
--------------------------------------------
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             20     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              6  RELEASEDIR
     10.80     113.00 us     113.00 us     113.00 us              1    GETXATTR
     28.68     150.00 us     139.00 us     161.00 us              2      STATFS
     60.52     158.25 us     117.00 us     226.00 us              4      LOOKUP
 
    Duration: 8629 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
. . .

You will receive a lot of information about each node with this command.

For a list of all of the GlusterFS associated components running on each of your nodes, you can type:

server1/server2

$ sudo gluster volume status

Console output:

Status of volume: volume1
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick server1.example.com:/gluster-storage             49152   Y       2808
Brick server2.example.com:/gluster-storage             49152   Y       2741
NFS Server on localhost                                 2049    Y       3271
Self-heal Daemon on localhost                           N/A     Y       2758
NFS Server on server1.example.com                      2049    Y       3211
Self-heal Daemon on server1.example.com                N/A     Y       2825

There are no active volume tasks

If you are going to be administering your GlusterFS storage volumes, it may be a good idea to drop into the GlusterFS console. This will allow you to interact with your GlusterFS environment without needing to type sudo gluster before everything:

server1/server2

$ sudo gluster

This will give you a prompt where you can type your commands. This is a good one to get yourself oriented:

> help

When you are finished, exit like this:

> exit

Conclusion

At this point, you should have a redundant storage system that will allow us to write to two separate servers simultaneously. This can be useful for a great number of applications and can ensure that our data is available even when one server goes down.

Leave a Reply

Your email address will not be published. Required fields are marked *