The point of no return

I already explained how to daemonize java applications on SysV-style system in here. Since CentOS7/RHEL7 comes with Systemd which is a system and service manager for Linux we migrate old init scripts to the new system.

Again we’ll use our best budy Daemonize but this time we gonna compile it from source because although it’s signed as approved on fedora package db, Daemonize is not in EPEL7 repository for now. I’ll not going to details how to compile install etc. but I assume that you installed daemonize into /usr/local/sbin

So we need to create two files /etc/sysconfig/fixtures and /lib/systemd/system/fixtures.service
This 1st file is where we define java releated variables such as user, java path, arguments, log files etc..

/etc/sysconfig/fixtures
1
2
3
4
5
6
7
8
9
10
# Configz for fixtures service
 
JAVA_USER="pronet"
JAVA_STDOUT="/var/log/pronet/fixtures.log"
JAVA_STDERR="/var/log/pronet/fixtures-error.log"
JAVA_BIN="/usr/java/jdk1.7.0_71/bin/java"
JAVA_APPDIR="/opt/pronet/fixtures"
ARG1="-Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties"
ARG2="-Dlog4j.configuration=file:/opt/pronet/fixtures/fixtures-log.properties"
ARG3="-jar /opt/pronet/fixtures/fixtures.jar"

2nd file is service file for fixtures where we define systemd releted variables. There are plenty of documents in Freedesktop Systemd wiki, if you want to know more about I advice you to read them. But roughly unit: consist information about a service, a socket, a device etc, service: information about a process controlled and supervised by systemd and install: installation information for the unit

/lib/systemd/system/fixtures.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[Unit]
Description=Fixtures Service
After=syslog.target
After=network.target
 
[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/fixtures
ExecStart=/usr/local/sbin/daemonize -u $JAVA_USER -o $JAVA_STDOUT -e $JAVA_STDERR -c $JAVA_APPDIR $JAVA_BIN $ARG1 $ARG2 $ARG3
ExecStop=/bin/kill -TERM $MAINPID
TimeoutSec=300
 
[Install]
WantedBy=multi-user.target

Let’s start and stop the service

[root@Srv25 pronet]# systemctl start fixtures
[root@Srv25 pronet]# systemctl stop fixtures

if there is something wrong all service files and docker containers insert data into the systemd journal and we can read the journal :)

[root@Srv25 pronet]# journalctl -u fixtures.service

Checking the service status

[root@Srv25 pronet]# systemctl status fixtures
fixtures.service - Fixtures Service
   Loaded: loaded (/usr/lib/systemd/system/fixtures.service; disabled)
   Active: active (running) since Wed 2014-10-29 21:21:49 EET; 13min ago
 Main PID: 28859 (java)
   CGroup: /system.slice/fixtures.service
           └─28859 /usr/java/jdk1.7.0_71/bin/java -Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties -Dlog4j.configuration=file:...
 
Oct 29 21:21:49 Srv25 systemd[1]: Started Fixtures Service.

Enable the service to be started on bootup

[root@Srv25 pronet]# systemctl enable fixtures.service
ln -s '/usr/lib/systemd/system/fixtures.service' '/etc/systemd/system/multi-user.target.wants/fixtures.service'
[root@Srv25 pronet]#

So that’s how it works..

We’re using Hsqldb and Glassfish on a project, our developers found some scripts to run java applications as daemon and it was ok for a while but when they started to report problems about the services I checked what they’re using. A lot of function’s, write’s, check’s, for’s, while’s, if’s, else’s etc.. There should be an easy way to do this? Guess what I decided to make it simple, Yet another NIH Syndrome ? :) Let me try to explain.. Bytheway we dont want to compile source code or add any 3rd party repositories.

But 1st what is a daemon? Daemon stands for Disk and Execution Monitor and is a long-running background process that answers requests for services. As explained in Wikipedia:

On a Unix-like system, the common method for a process to become a daemon, when the process is started from the command line or from a startup script such as an init script or a SystemStarter script, involves:

  1. Dissociating from the controlling tty
  2. Becoming a session leader
  3. Becoming a process group leader
  4. Executing as a background task by forking and exiting (once or twice). This is required sometimes for the process to become a session leader. It also allows the parent process to continue its normal execution.
  5. Setting the root directory (/) as the current working directory so that the process does not keep any directory in use that may be on a mounted file system (allowing it to be unmounted).
  6. Changing the umask to 0 to allow open(), creat(), et al. operating system calls to provide their own permission masks and not to depend on the umask of the caller
  7. Closing all inherited files at the time of execution that are left open by the parent process, including file descriptors 0, 1 and 2 for the standard streams (stdin, stdout and stderr). Required files will be opened later.
  8. Using a logfile, the console, or /dev/null as stdin, stdout, and stderr

There are some methods to run application as daemon on Linux such as; start-stop-daemon (part of dpkg package), Red Hat init.d function, Nohup

Start-stop-daemon: It’s a good tool but EPEL dpkg package does not consist it, I don’t know why ?
Red Hat init.d function: Setting application/working directory could be problem also not possible to get StdOut & StdErr outputs as log file
Nohup: No automation, you have to write lots of things.. nohup $command >>$log_file 2>&1 & echo \$! >$pid_file

No start-stop-daemon in EPEL repository, RHEL init.d daemon function is not enough, Nohup requires bash scripting knowledge.. Ok what now ??
Instead use Daemonize. Daemonize runs a command as a Unix daemon, it’s a good alternative for start-stop-daemon and daemonize package is in the EPEL repository.

I wrote an init.d script (Jdis) to run java applications as daemon with daemonize on RHEL and CentOS Linux, you can download Jdis from GitHub: java_daemon-init.sh

Hsqldb Init script example

An example from real world situation; Hsqldb. HyperSQL DataBase (Hsqldb) is a relational database software written in Java. It offers a small, fast multithreaded and transactional database engine with in-memory and disk-based tables and supports embedded and server mode.

Let’s daemonize it..

We dont want to run the application with root user

[root@zion ~]# useradd hsqldb -s /sbin/nologin

Change owner of the application directory

[root@zion ~]# chown -R hsqldb:hsqldb /opt/hsqldb
/etc/init.d/hsqldb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
#!/bin/bash
#
# Jdis - Java daemon init-script for Red Hat / CentOS Linux
# Ali Erdinc Koroglu - http://ae.koroglu.org
# License : GNU GPL (http://www.gnu.org/licenses/gpl.html)
#
# You must install daemonize package from EPEL repository to use this script.
# How to add EPEL repository: http://ae.koroglu.org/docs/adding-epel-repository-on-centos/
#
# History:
# 2014-08-19 : First release
 
# chkconfig: 345 85 15
# description: Java daemon script
 
### BEGIN INIT INFO
# Provides:
# Required-Start: $local_fs $network $syslog $time
# Required-Stop: $local_fs $network $syslog $time
# Short-Description: start and stop Java daemons
# Description: Java daemon init script
### END INIT INFO
 
# source function library
. /etc/init.d/functions
 
# Java Home
java_home="/usr/java/jdk1.7.0_45"                                               # java path
 
# Service settings
service_name="hsqldb"                                                           # Service name
service_user="hsqldb"                                                           # User/group of process
pid_file="/var/run/$service_name.pid"                                           # Pid file
log_file="/var/log/$service_name/$service_name.log"                             # StdOut log file
errlog_file="/var/log/$service_name/$service_name-error.log"                    # StdErr log file
java="$java_home/bin/java"                                                      # Java binary
java_appdir="/opt/hsqldb/data"                                                  # Application path
java_applibdir="/opt/hsqldb/lib"                                                # Application Lib path
java_arg1="-server -Xms1G -Xmx6G -XX:NewSize=256m"                              # Argument 1
java_arg2="-XX:MaxNewSize=256m -XX:PermSize=512m -XX:MaxPermSize=512m"          # Argument 2
java_arg3="-Dproject.properties=$java_appdir/test.properties"                   # Argument 3
java_arg4="-classpath $java_applibdir/hsqldb.jar org.hsqldb.server.Server"      # Argument 4
java_args="$java_arg1 $java_arg2 $java_arg3 $java_arg4"                         # Java Arguments
 
RETVAL=0
start() {
    [ -x $java ] || exit 5
    echo -n $"Starting $service_name: "
    if [ $EUID -ne 0 ]; then
        RETVAL=1
        failure
    elif [ -s /var/run/$service_name.pid ]; then
        RETVAL=1
        echo -n $"already running.."
        failure
    else
        daemonize -u $service_user -p $pid_file -o $log_file -e $errlog_file -c $java_appdir $java $java_args && success || failure
        RETVAL=$?
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$service_name
    fi;
    echo
    return $RETVAL
}
 
stop() {
    echo -n $"Stopping $service_name: "
    if [ $EUID -ne 0 ]; then
        RETVAL=1
        failure
    else
        killproc -p $pid_file
        RETVAL=$?
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$service_name
    fi;
    echo
    return $RETVAL
}
 
restart(){
    stop
    start
}
 
case "$1" in
    start)
        start
        RETVAL=$?
        ;;
    stop)
        stop
        RETVAL=$?
        ;;
    restart)
        restart
        RETVAL=$?
        ;;
    status)
        status $service_name
        RETVAL=$?
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart}"
        RETVAL=2
esac
exit $RETVAL

lets check..

[root@zion ~]# /etc/init.d/hsqldb start
Starting hsqldb:                                           [  OK  ]
[root@zion ~]# ps aux | grep hsqldb
hsqldb     463  178  1.0 8555172 349364 ?      Ssl  15:04   0:10 /usr/java/jdk1.7.0_45/bin/java -server -Xms1G -Xmx6G -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=512m -XX:MaxPermSize=512m -Dproject.properties=/usr/share/pronet/hsqldb/data/test.properties -classpath /usr/share/pronet/hsqldb/lib/hsqldb.jar org.hsqldb.server.Server
root       485  0.0  0.0 103244   848 pts/0    R+   15:04   0:00 grep hsqldb
[root@zion ~]# /etc/init.d/hsqldb status
hsqldb (pid  463) is running...
[root@zion ~]# /etc/init.d/hsqldb stop
Stopping hsqldb:                                           [  OK  ]