Author: Ngu Nguyen

RabbitMQ installation guide on Linux Ubuntu 16.04

RabbitMQ installation guide on Linux Ubuntu 16.04

Introduction

With more than 35,000 production deployments of RabbitMQ world-wide at small startups and large enterprises, RabbitMQ is the most popular open source message broker.
RabbitMQ is lightweight and easy to deploy on premise and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

In this guide, we’ll cover how to install and configure a 3-node rabbitMQ cluster, with HA proxy and management plugin.

Installing RabbitMQ on single node

Update our system’s default application toolset

$ sudo apt-get update
$ sudo apt-get -y upgrade

Enable RabbitMQ appication repository

$ echo "deb http://www.rabbitmq.com/debian/ testing main" >> /etc/apt/sources.list

Add the verification key for the package

$ curl http://www.rabbitmq.com/rabbitmq-signing-key-public.asc | sudo apt-key add -

Update the source with our new addition from above

$ sudo apt-get update

And finally, download and install RabbitMQ

$ sudo apt-get install rabbitmq-server

Start rabbitMQ server

$ sudo systemctl restart rabbitmq-server.service

Congratulation!! You’ve just successfully install your rabbitMQ server.

Configure multi-cluster RabbitMQ

In this part, we’ll go through how to setup a multi-node RabbitMQ cluster system (3 nodes in this example).

Sync the erlang cookie

RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to determine whether they are allowed to communicate with each other. For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie. The cookie is just a string of alphanumeric characters. It can be as long or short as you like. Every cluster node must have the same cookie.

Erlang VM will automatically create a random cookie file when the RabbitMQ server starts up. The easiest way to proceed is to allow one node to create the file, and then copy it to all the other nodes in the cluster.

On Unix systems, the cookie will be typically located in /var/lib/rabbitmq/.erlang.cookie or $HOME/.erlang.cookie.

When the cookie is misconfigured (for example, not identical), RabbitMQ will log errors such as “Connection attempt from disallowed node” and “Could not auto-cluster”.

Configure hostname and RabbitMQ Nodename

Configure Hostname

Edit hosts:

$ sudo vim /etc/hosts
xxx.xxx.xxx.xxx rabbit01
xxx.xxx.xxx.xxx rabbit02
xxx.xxx.xxx.xxx rabbit03

Configure RabbitMQ hostname

Edit rabbitmq-evn.conf:

rabbit01$ sudo vim /etc/rabbitmq/rabbitmq-env.conf
NODENAME=rabbit@rabbit01
rabbit02$ sudo vim /etc/rabbitmq/rabbitmq-env.conf
NODENAME=rabbit@rabbit02
rabbit03$ sudo vim /etc/rabbitmq/rabbitmq-env.conf
NODENAME=rabbit@rabbit03

Staring independent nodes

Clusters are set up by re-configuring existing RabbitMQ nodes into a cluster configuration. Hence the first step is to start RabbitMQ on all nodes in the normal way:

rabbit01$ rabbitmq-server -detached
rabbit02$ rabbitmq-server -detached
rabbit03$ rabbitmq-server -detached

This creates three independent RabbitMQ brokers, one on each node, as confirmed by the cluster_status command:

rabbit01$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit01 ...
[{nodes,[{disc,[rabbit@rabbit01]}]},{running_nodes,[rabbit@rabbit01]}]
...done.
rabbit02$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit02 ...
[{nodes,[{disc,[rabbit@rabbit02]}]},{running_nodes,[rabbit@rabbit02]}]
...done.
rabbit03$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit03 ...
[{nodes,[{disc,[rabbit@rabbit03]}]},{running_nodes,[rabbit@rabbit03]}]
...done.

The node name of a RabbitMQ broker started from the rabbitmq-server shell script is rabbit@shorthostname, where the short node name is lower-case (as in rabbit@rabbit01, above). If you use the rabbitmq-server.bat batch file on Windows, the short node name is upper-case (as in rabbit@RABBIT01). When you type node names, case matters, and these strings must match exactly.

Create the cluster

In order to link up our three nodes in a cluster, we tell two of the nodes, say rabbit@rabbit2 and rabbit@rabbit3, to join the cluster of the third, say rabbit@rabbit1.

We first join rabbit@rabbit2 in a cluster with rabbit@rabbit1. To do that, on rabbit@rabbit2 we stop the RabbitMQ application and join the rabbit@rabbit1 cluster, then restart the RabbitMQ application. Note that joining a cluster implicitly resets the node, thus removing all resources and data that were previously present on that node.

rabbit02$ rabbitmqctl stop_app
Stopping node rabbit@rabbit02 ...done.
rabbit02$ rabbitmqctl join_cluster rabbit@rabbit01
Clustering node rabbit@rabbit02 with [rabbit@rabbit01] ...done.
rabbit02$ rabbitmqctl start_app
Starting node rabbit@rabbit02 ...done.

We can see that the two nodes are joined in a cluster by running the cluster_status command on either of the nodes:

rabbit01$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit1 ...
[{nodes,[{disc,[rabbit@rabbit01,rabbit@rabbit02]}]},
{running_nodes,[rabbit@rabbit02,rabbit@rabbit01]}]
...done.
rabbit02$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit02 ...
[{nodes,[{disc,[rabbit@rabbit01,rabbit@rabbit02]}]},
{running_nodes,[rabbit@rabbit01,rabbit@rabbit02]}]
...done.

Now we join rabbit@rabbit03 to the same cluster. The steps are identical to the ones above, except this time we’ll cluster to rabbit02 to demonstrate that the node chosen to cluster to does not matter – it is enough to provide one online node and the node will be clustered to the cluster that the specified node belongs to.

rabbit03$ rabbitmqctl stop_app
Stopping node rabbit@rabbit03 ...done.
rabbit03$ rabbitmqctl join_cluster rabbit@rabbit02
Clustering node rabbit@rabbit03 with rabbit@rabbit02 ...done.
rabbit03$ rabbitmqctl start_app
Starting node rabbit@rabbit03 ...done.

We can see that the three nodes are joined in a cluster by running the cluster_status command on any of the nodes:

rabbit01$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit01 ...
[{nodes,[{disc,[rabbit@rabbit01,rabbit@rabbit02,rabbit@rabbit03]}]},
{running_nodes,[rabbit@rabbit03,rabbit@rabbit02,rabbit@rabbit01]}]
...done.
rabbit02$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit02 ...
[{nodes,[{disc,[rabbit@rabbit01,rabbit@rabbit02,rabbit@rabbit03]}]},
{running_nodes,[rabbit@rabbit03,rabbit@rabbit01,rabbit@rabbit02]}]
...done.
rabbit03$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit03 ...
[{nodes,[{disc,[rabbit@rabbit03,rabbit@rabbit02,rabbit@rabbit01]}]},
{running_nodes,[rabbit@rabbit02,rabbit@rabbit01,rabbit@rabbit03]}]
...done.

By following the above steps we can add new nodes to the cluster at any time, while the cluster is running.

Management Plugin

Enable rabbitmq_management in everynode

$ sudo rabbitmq-plugins enable rabbitmq_management

Create admin user on one node, this admin user can be use for any node in cluster

$ sudo rabbitmqctl add_user <user> <password>
$ sudo rabbitmqctl set_user_tags <user> administrator
$ sudo rabbitmqctl set_permissions -p / <user> ".*" ".*" ".*"

Now you can access management website with this user through  xxx.xxx.xxx.xxx:15672
sreenshot 01 - RabbitMq management 01
screenshot 02 - RabbitMQ management 02
screenshot 03 - RabbitMQ management 03

Setup HAProxy

Install haproxy

$ sudo apt-get install haproxy

Verify if HAProxy is working

$ haproxy -v
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau 

Configure HAproxy for RabbitMQ and RabbitMQ_management

$ sudo vim /etc/haproxy/haproxy.cfg
defaults
...
listen rabbitmq
        bind 0.0.0.0:5672
        mode tcp
        balance roundrobin
        timeout client 3h
        timeout server 3h
        option clitcpka
        server rabbit01 xxx.xxx.xxx.xx1:5672 check inter 5s fall 3 rise 2
        server rabbit02 xxx.xxx.xxx.xx2:5672 check inter 5s fall 3 rise 2
        server rabbit02 xxx.xxx.xxx.xx3:5672 check inter 5s fall 3 rise 2
listen rabbitmq_management
        bind 0.0.0.0:15672
        mode tcp
        balance roundrobin
        server rabbit01 xxx.xxx.xxx.xx1:15672 check fall 3 rise 2
        server rabbit02 xxx.xxx.xxx.xx2:15672 check fall 3 rise 2
        server rabbit02 xxx.xxx.xxx.xx3:15672 check fall 3 rise 2
Push notification for web browser

Push notification for web browser

I. Introduction

Have you ever seen your friend messages pop-ups on your desktop? Have you ever seen a amazon or lazada’s sale campaign information show up on your desktop even when you are not browsing those website? Those pop-ups are called push notification.
Push notification is a message pop-ups on your client’s desktops / devices. Publishers can send notifications anytime they want and receiver don’t have to be browsing website or to be in apps to receive those notifications. So why push notification are used?

For client

Push notification provide convenience and value to client. They can receive valued information at anytime like hot news, sport score, traffic, weather, flight checking, discount items, sale campaign, etc.

For publisher

Push notification is one of the most direct way to speak to user. They don’t get caught in spam filter, don’t get forgotten in an inbox. They can remind user to use app or visit website, whether they are browsing your website, open your app or not. They can also be use to drive actions, such as:

  • Promotion product or offer to increase sale
  • Improve customer experience
  • Send transaction receipt right away
  • Drive user to other marketing channels
  • Convert unknown user to known customer

In this article, we’ll focus on push notification for web browser. We will figure out how it works and go through some code sample to create our very first push notification system.
Let’s begin!

II. How it works

1. Figure

Figure 01 - Push notification system work flow
Figure 01 – Push notification system work flow

2. Definition

First, we’ll go through some definition that will be used in this article

  • Notification: a message displayed to the user outside of the app’s normal UI (i.e., the browser)
  • Service-worker: a script that your browser runs in the background, separate from a web page, opening the door to features that don’t need a web page or user interaction.
  • Push Server: the middle server, receive request from your server and send it to clients when they are online

3. Concept

The above figure-01 explain almost everything about how Push Notification works. In this part, we are going to explain it a bit deeper.
We have 3 main part in push notification jobs: producer (your server), client (user), and push server (google GCM, google FCM, amazonSNS, etc.). Each part have their own roles:

  • Client: receive push request and pop up notification.
  • Server: manage client token, make push requests.
  • Push server: the middle server, this will receive push request from server and send it to client when they are online.

And this is how they works

  • Step 00: At first, When user browsing your website, you have to register an service-worker and ask user for permission to run service-worker. If permission is not granted, then there’s nothing we can do more. If permission granted, let’s go to next step
  • Step 01: Client register itself to push server with api key and sender id to identify which app and server can send notification request to worker.
  • Step 02: Push server receive register request from client and return a token to worker, this token is now present for service-worker.
  • Step 03: Client receive token from push server and send it to server, server will save it to database for later use.
  • Step 04: Server now had client token. From now, whenever server want to send notification to client, just send notification request to push server with client’s token to identity client and a private key to validate data.
  • Step 05: When push server receive notification request, they will store it and wait for client online to send it to client. When client receive the push notification, service-worker will pop-up notification.

III. Service worker

How can we show notification even when browser wasn’t on? Because the one who show notification is the service-worker. So what is service-worker and how it works. Let’s figure it out.

1. What is service-worker

A service worker is a script that your browser runs in the background, separate from a web page, opening the door to features that don’t need a web page or user interaction. Today, they already include features like push notifications and background sync. In the future, service workers will support other things like periodic sync or geofencing. The core feature discussed in this tutorial is the ability to intercept and handle network requests, including programmatically managing a cache of responses.
The reason this is such an exciting API is that it allows you to support offline experiences, giving developers complete control over the experience.

2. Concept

Figure 02 - Service worker life circle
Figure 02 – Service worker life circle

A service worker has a lifecycle that is completely separate from your web page.
To install a service worker for your site, you need to register it, which you do in your page’s JavaScript. Registering a service worker will cause the browser to start the service worker install step in the background.
Typically during the install step, you’ll want to cache some static assets. If all the files are cached successfully, then the service worker becomes installed. If any of the files fail to download and cache, then the install step will fail and the service worker won’t activate (i.e. won’t be installed). If that happens, don’t worry, it’ll try again next time. But that means if it does install, you know you’ve got those static assets in the cache.
When installed, the activation step will follow and this is a great opportunity for handling any management of old caches, which we’ll cover during the service worker update section.
After the activation step, the service worker will control all pages that fall under its scope, though the page that registered the service worker for the first time won’t be controlled until it’s loaded again. Once a service worker is in control, it will be in one of two states: either the service worker will be terminated to save memory, or it will handle fetch and message events that occur when a network request or message is made from your page.

3. Sampe Code

Because of security reason – you can just register/run service-worker in some enviroment like https or localhost, etc… So for testing code, we’re gonna use localhost, you can easily build a simple server with extension “web server for Chrome”. For more information, read this

Register service-worker

main.js

if (‘serviceWorker’ in navigator) {
    navigator.serviceWorker.register(‘/serviceworker.js’)
    .then (function (reg) {
        console.log(‘Serviceworker registered successfully with scope: ‘,  reg.scope)
    }
    .catch (function(err) {
        console.log(“Something wrong: “, err);
    });
}

serviceworker.js

self.addEventListener('install', function(event) {
    // Handle install event
    console.log('install');
});
self.addEventListener('activate', function(event) {
    // Handle activate event
    console.log('activate');
});

When you run register command, service worker will install then activate automatically.
screenshot - 01 - Serviceworker register
To knwow more about service-worker register process (first time / update), read this article

Grant permission from user

Now your service worker is install and activate. But you have to grant permission from user to do stuff like notification.
main.js

notification.requestPermission(function(status) {
    console.log('Notification permission status:', status);
});

screenshot - 02 - request permission
This is one time action, after user allow or block, setting will save in notification exception
screenshot - 03 - permission result

Sumary code

main.js

navigator.serviceWorker.register('/serviceworker.js').then(function (reg) {
    console.log(reg);
})
switch (Notification.permission) {
    case 'granted':
        console.log('already granted');
        break;
    case 'denied':
        console.log('blocked!!')
        break;
    case 'default':
    default:
        console.log('get permission')
        Notification.requestPermission(function(status) {
            console.log('Notification permission status:', status);
        });
        break;
};

serviceworker.js

self.addEventListener('install', function(event) {
    // Handle install event
    console.log('install');
});
self.addEventListener('activate', function(event) {
    // Handle activate event
    console.log('activate');
});

IV. SIMPLE NOTIFICATION (WITHOUT PUSH SERVER)

In this section, we will go through how to show notification directly from your website.

1. Concept

Figure 03 - Simple notification work flow
Figure 03 – Simple notification work flow

Like we discussed above, the one who pop-ups notification is service-worker. So, when you have a activated and permission granted service-worker, you now can command service-worker pop-ups notification. You can perform this command directly from your website, and yes, of course, when you perform this way, you can only pop-ups notification when user are browsing your website. Let’s go through some sample code!

2. Sample Code

Command from main.js

main.js

function displayNotification() {
    if (Notification.permission == 'granted') {
        navigator.serviceWorker.getRegistration()
        .then(function(reg) {
            reg.showNotification('Hello world!');
        })
    }
}
displayNotification();

Command from service-worker itself

E.g. Auto show notification when service-worker install

// Listen for install event, set callback
self.addEventListener('install', function(event) {
    // Perform some task
    console.log('install');
    self.registration.showNotification("installed");
});

V. NOTIFICATION WITH PUSH SERVER (FCM)

In this section, we will go through how full push notification system works.

1. Concept

Figure 01 - Push notification system work flow
Figure 01 – Push notification system work flow

In this tutorial, we’ll use FCM push server, FCM push server is an Google service, it’s later version of GCM. For more information, read this

2. Sample Code

a. Create FCM app

First, we register our app to FCM

Go to this website: https://console.firebase.google.com
Add project

screenshot - 04 - FCM add project - 01
screenshot - 04 - FCM add project - 02

Now let’s store our information

Public key(info)
screenshot - 05 - FCM public info
Server(private) key
screenshot - 06 - FCM private key
You can use server key or legacy server key to validate your push message request. I usually use the short one (legacy server key).

b. Register user to push server, get returned token

main.js

importScripts("https://www.gstatic.com/firebasejs/4.1.3/firebase.js");

// Initialize Firebase
var config = {
    apiKey: "...",
    authDomain: "...firebaseapp.com",
    databaseURL: "https://...firebaseio.com",
    projectId: "...",
    storageBucket: "...",
    messagingSenderId: "..."
};
firebase.initializeApp(config);

// Get token
const messaging = firebase.messaging();

messaging.requestPermission()
.then(function() {
    	console.log("Permission granted");
    	return messaging.getToken();
})
.then(function(token) {
    	console.log(“Token: ”, token);
	// send token to your server here
})
.catch(function(err) {
    	console.log("Err: ", err);
})

// handle receive message
messaging.onMessage(function(payload) {
console.log("payload: ", payload);
})

Messaging is firebase’s message library. When you call messaging.requesPermission(). It will look for a serviceworker name ‘firebase-messaging-sw.js’ and then do the register stuffs. So we have to create our serviceworker to let those code run successfully.
firebase-messaging-sw.js

// welcome to firebase-messaging
    // do nothing

Run main.js and you will get something like this
screenshot - 07 - fcm get token
This token now present for your browser and serviceworker. Send those token to your server and we are ready to push notification.
There are 2 case when you receive request from push server: you’re focusing on website or not. If you are NOT focusing on website, the message will go to service worker and pop-ups notification. If you are focusing, it will trigger event ‘messaging.onMessage’ so you can handle freely, of course you can command service worker pop-ups notification but in my experience it will make user feel annoyed and distracted. Let’s try our magic!
Send notification from your server to fcm server

URL: https://fcm.googleapis.com/fcm/send
METHOD: POST
HEADER:
    Content-Type: application/json
    Authorization: key=<your secret key>
BODY:
    {
        "to": <service-worker's token>,
        "notification": {
            "title": "my title",
            "content": "my content"
        }
    }

When you name the data notification, service worker will pop-ups automatically if possible. Here the result,

Focusing: receive data

screenshot 08 - fcm msg receive on website

Un-Focusing: pop-up notification

screenshot 08 - fcm msg received on serviceworker
Congratulations!! Now you have learned how to push notification. You can now improve your website, increasing your user’s experiment.
In the next section, we will go a little bit deeper about FCM, how to full-control your push event.

VI. FCM ADVANCE – EVENT HANDLE

1. Custom service worker file location and name

Like we discussed above, when you use firebase message, they will automatically find the service worker name ‘firebase-messaging-sw.js’ at home base of your website. So what will we do when we want to keep it in another location or use another file name. Here the solutions, we’re gonna register a service worker first then make firebase use this serviceworker as their firebase-messaging-sw. And here the code:
main.js

var config = {
...
};
firebase.initializeApp(config);

const messaging = firebase.messaging();

navigator.serviceWorker.register('/path/to/your/service-worker/serviceworker.js')
.then(function(reg) {
    messaging.useServiceWorker(reg);

    messaging.requestPermission()
    .then(function() {
        console.log("Permission granted");
        return messaging.getToken();
    })
    .then(function(token) {
        console.log("Token: ", token);
    })
    .catch(function(err) {
        console.log("Err: ", err);
    })
})

2. Full-control received message

The firebase will pop-ups notification when you send notification command, but what if you want to handle data or content manually from service worker, here the solutions

  • When user focusing on your website, just like before, you can handle the message in onMessage event, so there’s nothing to do more here.
  • When user NOT focusing on your website, the message will be received by service worker so we’re gonna add message handle for service worker.

Let’s view the code:
serviceworker.js

messaging.setBackgroundMessageHandler(function(payload) {
    // do whatever you want here, in this example we pop-ups notification
    const title = payload.data.title;
    const options = {
        body: payload.data.content
    };
    return self.registration.showNotification(title, options);
});

** messaging.setBackgoundMessageHangler is just like messaging.onMessage event for serviceworker.

3. Handle token change

Sometimes, FCM change token of client, it’s not very critical but user will can’t receive message until next time they register the service-worker. So if you want to update token immediately, you have to handle those event. Here the code:
serviceworker.js

// Callback fired if Instance ID token is updated.
messaging.onTokenRefresh(function() {
    messaging.getToken()
    .then(function(refreshedToken) {
        console.log('Token refreshed.');
	// send new token info to server here
    })
    .catch(function(err) {
        console.log('Unable to retrieve refreshed token ', err);
    });
});

4. Send message to group of users

This is the solutions of sending bulk push message. The idea is adding user’s tokens into groups. Whenever you want to send all message to user in that group, you just have to send only one message. Let’s figure it out.

Register token to group

Register one token
URL: https://iid.googleapis.com/iid/v1/<user’s Token>/rel/topics/<Topic name>
METHOD: POST
HEADER: 
    Content-Type: application/json
    Authorization: key=<your secret key>
Bulk register
URL: https://iid.googleapis.com/iid/v1:batchAdd
METHOD: POST
HEADER:
    Content-Type: application/json
    Authorization: key=<your secret key>
    Cache-Control: no-cache
BODY:
    {
        "to": "/topics/<TOPIC NAME>",
        "registration_tokens": <"Token1", “Token2”,...>
    }

Send push message to group

URL: https://fcm.googleapis.com/fcm/send
METHOD: POST
HEADER:
    Authorization: key=<your server secret key>
    Content-Type: application/json
BODY
    {
        "to": "/topics/<TOPIC NAME>",
        "notification": {
            "title": "my title",
            "body": "my content"
        }
    }

For more information: read this

VII. NOTIFICATION – STRUCTURE AND EVENT HANDLE

Uptil now in this article, we just pop-ups the most simple notification with title and body. In this section, we will talk about all elements of an notification, and how we let them do some stuff like boozing, or open new tab onclick, etc. Let’s begin!

1. Notification structure

This is the API of showing a notification

(ServiceWorkerRegistration).showNotification(<title>, <options>);

The <title> is a string, and the <options> can be any of the following:

{
  	"//": "Visual Options",
  	"body": "<String>",
  	"icon": "<URL String>",
  	"image": "<URL String>",
  	"badge": "<URL String>",
  	"vibrate": "<Array of Integers>",
  	"sound": "<URL String>",
  	"dir": "<String of 'auto' | 'ltr' | 'rtl'>",

  	"//": "Behavioural Options",
  	"tag": "<String>",
  	"data": "<Anything>",
  	"requireInteraction": "<boolean>",
  	"renotify": "<Boolean>",
  	"silent": "<Boolean>",

  	"//": "Both Visual & Behavioural Options",
  	"actions": "<Array of Strings>",

  	"//": "Information Option. No visual affect.",
  	"timestamp": "<Long>"
}

2. Event handle

We will go through some important notification event

Install event: trigger after install

serviceworker.js

// Listen for install event, set callback
self.addEventListener('install', function(event) {
    // Perform some task
    console.log('install');
});

Active event: trigger after active

serviceworker.js

self.addEventListener('activate', function(event) {
    // Perform some task
    console.log('activate');
});

Close event: trigger after close

serviceworker.js

self.addEventListener('notificationclose', function(event) {
    // Perfrom some task
    console.log('Closed notification’);
})

Click event: trigger after click

serviceworker.js

self.addEventListener('notificationclick', function(event) {
    var notification = event.notification;
    var action = event.action;
    switch (action) {
        case <action-name>:  	// define in options->action
            break;
        default:		// unrecognized actions (click on body)
            break;
    }
})

** There are also push event, but since we use firebase messaging, we use message.onMessage or message.setBackgroundMessageHandler instead.