How to use nohup to execute commands in the background and keep it running after you exit from a shell promt

By Tan Nguyen | July 17, 2017 | Comments 0 Comment

Most of the time you login into remote server via ssh. If you start a shell script or command and you exit (abort remote connection), the process/command will get killed. Sometimes a job or command takes a long time. If you are not sure when the job will finish, then it is better to leave job running in background. But, if you log out of the system, the job will be stopped and terminated by your shell. What do you do to keep job running in the background when process gets SIGHUP?

nohup command to the rescue

In these situations, we can use nohup command line-utility which allows to run command/process or shell script that can continue running in the background after we log out from a shell.

nohup is a POSIX command to ignore the HUP (hangup) signal. The HUP signal is, by convention, the way a terminal warns dependent processes of logout.

Output that would normally go to the terminal goes to a file called nohup.out if it has not already been redirected.

nohup command syntax:

The syntax is as follows

$ nohup command-name &
$ exit

$ nohup /path/to/command-name arg1 arg2 > myoutput.log &
$ exit

Where,

command-name: is name of shell script or command name. You can pass argument to command or a shell script.
&: nohup does not automatically put the command it runs in the background; you must do that explicitly, by ending the command line with an & symbol.
exit: after nohup and & start the command in the background, you will still have to type exit or CTRL-D to go back to bash cursor.

Use jobs -l command to list all jobs:

$ jobs -l

nohup command examples

First, login to remote server using ssh command:

$ ssh user@remote.server.com

I am going to execute a shell script called pullftp.sh:

$ nohup pullftp.sh &

Type exit or press CTRL + D to exit from remote server:

> exit

In this example, I am going to find all programs and scripts with setuid bit set on, enter:

$ nohup find / -xdev -type f -perm +u=s -print > out.txt &

Type exit or press CTRL + D to exit from remote server. The job still keeps on running after you exit.

> exit

nohup is often used in combination with the nice command to run processes on a lower priority.

Please note that nohup does not change the scheduling priority of COMMAND; use nice command for that purpose. For example:

$ nohup nice -n -5 ls / > out.txt &

As you can see nohup keep processes running after you exit from a shell. Read man page of nohup(1) and nice(1) for more information. Please note that nohup is almost available on Solaris/BSD/Linux/UNIX variants.

What’s the difference between nohup and ampersand (&)

Both nohup myprocess.out & or myprocess.out & set myprocess.out to run in the background. After I shutdown the terminal, the process is still running. What’s the difference between them?

nohup catches the hangup signal (see man 7 signal) while the ampersand doesn’t (except the shell is confgured that way or doesn’t send SIGHUP at all).

Normally, when running a command using & and exiting the shell afterwards, the shell will terminate the sub-command with the hangup signal (kill -SIGHUP <pid>). This can be prevented using nohup, as it catches the signal and ignores it so that it never reaches the actual application.

In case you’re using bash, you can use the command shopt | grep hupon to find out whether your shell sends SIGHUP to its child processes or not. If it is off, processes won’t be terminated, as it seems to be the case for you. More information on how bash terminates applications can be found here.

There are cases where nohup does not work, for example when the process you start reconnects the SIGHUP signal, as it is the case here.

Advanced topics

Existing jobs, processes

Some shells (e.g. bash) provide a shell builtin that may be used to prevent SIGHUP being sent or propagated to existing jobs, even if they were not started with nohup. In bash, this can be obtained by using disown -h job; using the same builtin without arguments removes the job from the job table, which also implies that the job will not receive the signal. Before using disown on an active job, it should be stopped by Ctrl-Z, and continued in the background by the bg command.^[2] Another relevant bash option is shopt huponexit, which automatically sends the HUP signal to jobs when the shell is exiting normally.^[3]

The AIX and Solaris versions of nohup have a -p option that modifies a running process to ignore future SIGHUP signals. Unlike the above-described disown builtin of bash, nohup -p accepts process IDs.^[4]

Overcoming hanging

Note that nohupping backgrounded jobs is typically used to avoid terminating them when logging off from a remote SSH session. A different issue that often arises in this situation is that ssh is refusing to log off (“hangs”), since it refuses to lose any data from/to the background job(s).^[5]^[6] This problem can also be overcome by redirecting all three I/O streams:

$ nohup ./myprogram > foo.out 2> foo.err < /dev/null &

Also note that a closing SSH session does not always send a HUP signal to depending processes. Among others, this depends on whether a pseudo-terminal was allocated or not.^[7]

Alternatives

Terminal multiplexer

A terminal multiplexer can run a command in a separate session, detached from the current terminal, which means that if the current session ends, the detached session and its associated processes keeps running. One can then reattach to the session later on.

For example, the following invocation of screen will run somescript.sh in the background of a detached session:

screen -A -m -d -S somename ./somescript.sh &

disown command

The disown command is used to remove jobs from the job table, or to mark jobs so that a SIGHUP signal is not sent on session termination.

Here is how you can try it out:

$ pullftp.sh &
$ disown -h
$ exit

From the bash bash(1) man page:

By default, removes each JOBSPEC argument from the table of active jobs. If the -h option is given, the job is not removed from the table, but is marked so that SIGHUP is not sent to the job if the shell receives a SIGHUP. The -a option, when JOBSPEC is not supplied, means to remove all jobs from the job table; the -r option means to remove only running jobs.

at command

You can use at command to queue a job for later execution. For example, you can run pullftp.sh script to queue (one minute) later execution:

$ echo "pullftp.sh" | at now + 1 minute