Tag: Linux

How to schedule tasks in Linux using crontab

How to schedule tasks in Linux using crontab

Introduction

Crontab is an utility in Linux that allow us to run jobs or commands on a specific schedule, just like Task Scheduler in Windows.

What is crontab

The crontab (cron derives from chronos, Greek for time; tab stands for table) command, found in Unix and Unix-like operating systems, is used to schedule commands to be executed periodically. To see what crontabs are currently running on your system, you can open a terminal and run:

sudo crontab -l

If you have never used it before, the system may tell you that there’s no crontab for your current user.

To edit the cronjobs, you can run

sudo crontab -e

How crontab works

Each user in Linux system has a file located at

/var/spool/cron/

containing a list of commands that will be executed on a timely basis.

The content of a crontab file looks like this:

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
0 */3 * * * /home/myname/scripts/my-script.sh

Notice the last line

0 */3 * * * /home/myname/scripts/my-script.sh

That line tells the system to run /home/myname/scripts/my-script.sh every 3 hours.

A crontab file can have multiple entries, each of which represents a cron job.

Note that this file is not intended to be edited directly, but rather by using the crontab command.

Don’t worry, we will go into detail how that works in a couple of minutes.

Structure of a crontab entry

As you can see in the last section, a crontab entry has a syntax like this:

# m h dom mon dow command
0 */3 * * * /home/myname/scripts/my-script.sh

For each entry, the following parameters must be included:

  1. m, a number (or list of numbers, or range of numbers), representing the minute of the hour (0-59);
  2. h, a number (or list of numbers, or range of numbers), representing the hour of the day (0-23);
  3. dom, a number (or list of numbers, or range of numbers), representing the day of the month (1-31);
  4. mon, a number (or list, or range), or name (or list of names), representing the month of the year (1-12);
  5. dow, a number (or list, or range), or name (or list of names), representing the day of the week (0-7, 0 or 7 is Sunday); and
  6. command, which is the command to be run, exactly as it would appear on the command line.

A field may be an asterisk (*), which always stands for “first through last”.

Ranges of numbers are allowed. Ranges are two numbers separated with a hyphen. The specified range is inclusive; for example, 8-11 for an “hours” entry specifies execution at hours 8, 9, 10 and 11.

Lists are allowed. A list is a set of numbers (or ranges) separated by commas. Examples: “1,2,5,9“, “0-4,8-12“.

Step values can be used in conjunction with ranges. For example, “0-23/2” can be used in the hours field to specify command execution every other hour. Steps are also permitted after an asterisk, so if you want to say “every two hours”, you can use “*/2“.

Names can also be used for the “month” and “day of week” fields. Use the first three letters of the particular day or month (case doesn’t matter). Ranges or lists of names are not allowed.

The “sixth” field (the rest of the line) specifies the command to be run. The entire command portion of the line, up to a newline or % character, will be executed by /bin/sh or by the shell specified in the SHELL variable of the cronfile. Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be sent to the command as standard input.

Note that the day of a command’s execution can be specified by two fields: day of month, and day of week. If both fields are restricted (in other words, they aren’t *), the command will be run when either field matches the current time. For example, “30 4 1,15 * 5” would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.

How to use crontab examples

List current crontab

crontab -l

Edit your crontab

crontab -e

This command will open up the crontab in a editor (usually vi or vim). Edit and save your cron jobs as you wanted.

Remove your crontab (un-scheduling all crontab jobs)

crontab -r

Back up your crontab to my-crontab file

crontab -l > my-crontab

Load your crontab from my-crontab file

crontab my-crontab

This does the same syntax checking as crontab -e.

Edit the crontab of the user named “charles”

sudo crontab -u charles -e

Note that the -u option requires administrator privileges, so that command is executed using sudo.

View the crontab of user “jeff”

sudo crontab -u jeff -l

Remove the crontab of user “sandy”

sudo crontab -u sandy -r

Check if your crontab is working

If you want to check if your cron job is executed as expected, or when it was last run, check the crontab log as following.

On a default installation the cron jobs get logged to

/var/log/syslog

You can see just cron jobs in that logfile by running

grep CRON /var/log/syslog

If you haven’t reconfigured anything, the entries will be in there.

Examples of crontab entries

Run the shell script /home/melissa/backup.sh on January 2 at 6:15 A.M.

15 6 2 1 * /home/melissa/backup.sh

Same as the above entry. Zeroes can be added at the beginning of a number for legibility, without changing their value.

15 06 02 Jan * /home/melissa/backup.sh

Run /home/carl/hourly-archive.sh every hour, on the hour, from 9 A.M. through 6 P.M., every day.

0 9-18 * * * /home/carl/hourly-archive.sh

Run /home/wendy/script.sh every Monday, at 9 A.M. and 6 P.M.

0 9,18 * * Mon /home/wendy/script.sh

Run /usr/local/bin/backup at 10:30 P.M., every weekday.

30 22 * * Mon,Tue,Wed,Thu,Fri /usr/local/bin/backup

Run /home/bob/script.sh every 3 hours.

0 */3 * * * /home/bob/script.sh

Run /home/mirnu/script.sh every minute (of every hour, of every day of the month, of every month and every day in the week).

* * * * * /home/mirnu/script.sh

Run /home/mirnu/script.sh 10 minutes past every hour on the 1st of every month.

10 * 1 * * /home/mirnu/script.sh

Special keywords

For the first (minute) field, you can also put in a keyword instead of a number:

@reboot Run once, at startup
@yearly Run once a year "0 0 1 1 *
@annually (same as @yearly)
@monthly Run once a month "0 0 1 * *"
@weekly Run once a week "0 0 * * 0"
@daily Run once a day "0 0 * * *"
@midnight (same as @daily)
@hourly Run once an hour "0 * * * *"

Leaving the rest of the fields empty, this would be valid:

@daily /bin/execute/this/script.sh

Storing the crontab output

By default cron saves the output of /bin/execute/this/script.sh in the user’s mailbox (root in this case). But it’s prettier if the output is saved in a separate logfile. Here’s how:

*/10 * * * * /bin/execute/this/script.sh >> /var/log/script_output.log 2>&1

Explanation

Linux can report on different levels. There’s standard output (STDOUT) and standard errors (STDERR). STDOUT is marked 1, STDERR is marked 2. So the following statement tells Linux to store STDERR in STDOUT as well, creating one datastream for messages & errors:

2>&1

Now that we have 1 output stream, we can pour it into a file. Where > will overwrite the file, >> will append to the file. In this case we’d like to to append:

>> /var/log/script_output.log

Mailing the crontab output

By default cron saves the output in the user’s mailbox (root in this case) on the local system. But you can also configure crontab to forward all output to a real email address by starting your crontab with the following line:

MAILTO="yourname@yourdomain.com"

Mailing the crontab output of just one cronjob

If you’d rather receive only one cronjob’s output in your mail, make sure this package is installed:

$ aptitude install mailx

And change the cronjob like this:

*/10 * * * * /bin/execute/this/script.sh 2>&1 | mail -s "Cronjob ouput" yourname@yourdomain.com

Trashing the crontab output

Now that’s easy:

*/10 * * * * /bin/execute/this/script.sh > /dev/null 2>&1

Just pipe all the output to the null device, also known as the black hole. On Unix-like operating systems, /dev/null is a special file that discards all data written to it.

 Caveats

Many scripts are tested in a Bash environment with the PATH variable set. This way it’s possible your scripts work in your shell, but when run from cron (where the PATH variable is different), the script cannot find referenced executables, and fails.

It’s not the job of the script to set PATH, it’s the responsibility of the caller, so it can help to echo $PATH, and put PATH=<the result> at the top of your cron files (right below MAILTO).

How to setup Linux, Nginx, MySQL, PHP (LEMP stack) on Ubuntu 16.04

How to setup Linux, Nginx, MySQL, PHP (LEMP stack) on Ubuntu 16.04

Introduction

The LEMP software stack is a group of software that can be used to serve dynamic web pages and web applications. This is an acronym that describes a Linux operating system, with an Nginx web server. The backend data is stored in the MySQL database and the dynamic processing is handled by PHP.

In this guide, we will demonstrate how to install a LEMP stack on an Ubuntu 16.04 server. The Ubuntu operating system takes care of the first requirement. We will describe how to get the rest of the components up and running.

Prerequisites

Before you complete this tutorial, you should have a regular, non-root user account on your server with sudo privileges. You can learn how to set up this type of account by completing DigitalOcean’s Ubuntu 16.04 initial server setup.

Once you have your user available, sign into your server with that username. You are now ready to begin the steps outlined in this guide.

Step 1: Install the Nginx Web Server

In order to display web pages to our site visitors, we are going to employ Nginx, a modern, efficient web server.

All of the software we will be using for this procedure will come directly from Ubuntu’s default package repositories. This means we can use the apt package management suite to complete the installation.

Since this is our first time using apt for this session, we should start off by updating our local package index. We can then install the server.

Install nginx

sudo apt-get update
sudo apt-get install nginx

On Ubuntu 16.04, Nginx is configured to start running upon installation.

If you are have the ufw firewall running, as outlined in our initial setup guide, you will need to allow connections to Nginx. Nginx registers itself with ufw upon installation, so the procedure is rather straight forward.

It is recommended that you enable the most restrictive profile that will still allow the traffic you want. Since we haven’t configured SSL for our server yet, in this guide, we will only need to allow traffic on port 80.

Enable ufw to allow HTTP

sudo ufw allow 'Nginx HTTP'

Verify the change

sudo ufw status

You should see HTTP traffic allowed in the displayed output:

Output
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
Nginx HTTP                 ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

With the new firewall rule added, you can test if the server is up and running by accessing your server’s domain name or public IP address in your web browser.

If you do not have a domain name pointed at your server and you do not know your server’s public IP address, you can find it by typing one of the following into your terminal:

Find server’s public IP address

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'

This will print out a few IP addresses. You can try each of them in turn in your web browser.

As an alternative, you can check which IP address is accessible as viewed from other locations on the internet:

Alternative way to find server’s public IP address

curl -4 icanhazip.com

Type one of the addresses that you receive in your web browser. It should take you to Nginx’s default landing page:

Test Nginx’s default landing page in browser

http://server_domain_or_IP


If you see the above page, you have successfully installed Nginx.

Step 2: Install MySQL to Manage Site Data

Now that we have a web server, we need to install MySQL, a database management system, to store and manage the data for our site.

Install MySQL server

sudo apt-get install mysql-server

You will be asked to supply a root (administrative) password for use within the MySQL system.

The MySQL database software is now installed, but its configuration is not exactly complete yet.

To secure the installation, we can run a simple security script that will ask whether we want to modify some insecure defaults. Begin the script by typing:

Secure MySQL installation

sudo mysql_secure_installation

You will be asked to enter the password you set for the MySQL root account. Next, you will be asked if you want to configure the VALIDATE PASSWORD PLUGIN.

Warning: Enabling this feature is something of a judgment call. If enabled, passwords which don’t match the specified criteria will be rejected by MySQL with an error. This will cause issues if you use a weak password in conjunction with software which automatically configures MySQL user credentials, such as the Ubuntu packages for phpMyAdmin. It is safe to leave validation disabled, but you should always use strong, unique passwords for database credentials.

Answer y for yes, or anything else to continue without enabling.

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No:

If you’ve enabled validation, you’ll be asked to select a level of password validation. Keep in mind that if you enter 2, for the strongest level, you will receive errors when attempting to set any password which does not contain numbers, upper and lowercase letters, and special characters, or which is based on common dictionary words.

There are three levels of password validation policy:

LOW    Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary                  file

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 1

If you enabled password validation, you’ll be shown a password strength for the existing root password, and asked you if you want to change that password. If you are happy with your current password, enter n for “no” at the prompt:

Using existing password for root.

Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : n

For the rest of the questions, you should press Y and hit the Enter key at each prompt. This will remove some anonymous users and the test database, disable remote root logins, and load these new rules so that MySQL immediately respects the changes we have made.

At this point, your database system is now set up and we can move on.

Step 3: Install PHP for Processing

We now have Nginx installed to serve our pages and MySQL installed to store and manage our data. However, we still don’t have anything that can generate dynamic content. We can use PHP for this.

Since Nginx does not contain native PHP processing like some other web servers, we will need to install php-fpm, which stands for “fastCGI process manager”. We will tell Nginx to pass PHP requests to this software for processing.

We can install this module and will also grab an additional helper package that will allow PHP to communicate with our database backend. The installation will pull in the necessary PHP core files. Do this by typing:

Install php-fpm and php-mysql

sudo apt-get install php-fpm php-mysql

Configure the PHP Processor

We now have our PHP components installed, but we need to make a slight configuration change to make our setup more secure.

Open the main php-fpm configuration file with root privileges:

sudo nano /etc/php/7.0/fpm/php.ini

What we are looking for in this file is the parameter that sets cgi.fix_pathinfo. This will be commented out with a semi-colon (;) and set to “1” by default.

This is an extremely insecure setting because it tells PHP to attempt to execute the closest file it can find if the requested PHP file cannot be found. This basically would allow users to craft PHP requests in a way that would allow them to execute scripts that they shouldn’t be allowed to execute.

We will change both of these conditions by uncommenting the line and setting it to “0” like this:

Change cgi.fix_pathinfo

# /etc/php/7.0/fpm/php.ini

cgi.fix_pathinfo=0

Save and close the file when you are finished.

Now, we just need to restart our PHP processor by typing:

Restart PHP processor

sudo systemctl restart php7.0-fpm

This will implement the change that we made.

Step 4: Configure Nginx to Use the PHP Processor

Now, we have all of the required components installed. The only configuration change we still need is to tell Nginx to use our PHP processor for dynamic content.

We do this on the server block level (server blocks are similar to Apache’s virtual hosts). Open the default Nginx server block configuration file by typing:

Open nginx default configuration file

sudo nano /etc/nginx/sites-available/default

Currently, with the comments removed, the Nginx default server block file looks like this:

# /etc/nginx/sites-available/default

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;

    server_name _;

    location / {
        try_files $uri $uri/ =404;
    }
}

We need to make some changes to this file for our site.

  • First, we need to add index.php as the first value of our index directive so that files named index.php are served, if available, when a directory is requested.
  • We can modify the server_name directive to point to our server’s domain name or public IP address.
  • For the actual PHP processing, we just need to uncomment a segment of the file that handles PHP requests by removing the pound symbols (#) from in front of each line. This will be the location ~\.php$ location block, the included fastcgi-php.conf snippet, and the socket associated with php-fpm.
  • We will also uncomment the location block dealing with .htaccess files using the same method. Nginx doesn’t process these files. If any of these files happen to find their way into the document root, they should not be served to visitors.

The changes that you need to make are in bold in the text below:

# /etc/nginx/sites-available/default

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;
    index index.php index.html index.htm index.nginx-debian.html;

    server_name server_domain_or_IP;

    location / {
        try_files $uri $uri/ =404;
    }
    
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

    location ~ /\.ht {
        deny all;
    }
}

When you’ve made the above changes, you can save and close the file.

Test your configuration file for syntax errors by typing:

Test nginx configurations

sudo nginx -t

If any errors are reported, go back and recheck your file before continuing.

When you are ready, reload Nginx to make the necessary changes:

Reload nginx

sudo systemctl reload nginx

Step 5: Create a PHP File to Test Configuration

Your LEMP stack should now be completely set up. We can test it to validate that Nginx can correctly hand .php files off to our PHP processor.

We can do this by creating a test PHP file in our document root. Open a new file called info.php within your document root in your text editor:

Create a new PHP file to test PHP installation

sudo nano /var/www/html/info.php

Type or paste the following lines into the new file. This is valid PHP code that will return information about our server:

# /var/www/html/info.php

<?php
phpinfo();

When you are finished, save and close the file.

Now, you can visit this page in your web browser by visiting your server’s domain name or public IP address followed by /info.php:

http://server_domain_or_IP/info.php

You should see a web page that has been generated by PHP with information about your server:

f you see a page that looks like this, you’ve set up PHP processing with Nginx successfully.

After verifying that Nginx renders the page correctly, it’s best to remove the file you created as it can actually give unauthorized users some hints about your configuration that may help them try to break in. You can always regenerate this file if you need it later.

For now, remove the file by typing:

sudo rm /var/www/html/info.php

Conclusion

You should now have a LEMP stack configured on your Ubuntu 16.04 server. This gives you a very flexible foundation for serving web content to your visitors.