The point of no return

Centralized Supervisor Interface: Cesi

28th December 2014 by Ali Erdinç Köroğlu

I already mentioned about supervisor in here. But how about to manage all supervisors from one web interface which has authorization and advanced process management filtering? As you know Supervisord provides a basic web interface to monitor and restart your processes where supervisor installed but XML-RPC interface and API allows you to control remote supervisor and the programs it runs. So it’s possible.. One UI to rule them all..

Here is Cesi (Centralized supervisor interface): it’s a web interface to manage supervisors from single UI, developed by Gülşah Köse and Kaan Özdinçer.

Application Dependencies

  1. Python : a programming language :)
  2. Flask : a microframework for Python
  3. SQlite : a self-contained, serverless, zero-configuration, transactional SQL database engine

I’ll cover everything step by step for CentOS7 minimal installation..
Since Flask is not in CentOS7 or EPEL repositories, I’ll install it via pip. So what is pip? Pip is a tool for installing and managing Python packages from Python Package Index repository. We also need Git (for cesi) and Nginx (you can get nginx from EPEL or from official Nginx repository)

Required CentOS packages

[root@supervisord ~]# yum install python-pip sqlite git nginx

Flask installation

[root@supervisord ~]# pip install flask

Getting a clone from github

[root@supervisord ~]# cd /opt/
[root@supervisord opt]# git clone https://github.com/Gamegos/cesi

Initial database operations

[root@supervisord cesi]# sqlite3 /opt/cesi/cesi/userinfo.db < userinfo.sql

Conf file should be in /etc

[root@supervisord cesi]# mv /opt/cesi/cesi/cesi.conf /etc

Let us define some remote supervisors

/etc/cesi.conf
[node:srv4]
username = superv
password = superv1q2w3e
host = 192.168.9.4
port = 9001
 
[node:srv10]
username = superv
password = superv
host = 192.168.9.10
port = 9001
 
[environment:glassfish]
members = srv4, srv10
 
[cesi]
database = /root/cesi/cesi/userinfo.db
activity_log = /root/cesi/cesi/cesi_activity.log

Since cesi will run by nginx user /opt/cesi should be accessible for it

[root@supervisord opt]# chown -R nginx:nginx /opt/cesi

We’ll run cesi via supervisord :)

/etc/supervisor.d/cesi.ini
[program:cesi]
command=/bin/python /opt/cesi/cesi/web.py
process_name=%(program_name)s
user=nginx
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/cesi-stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/var/log/cesi-stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

Flask will initialise twice the application in debug mode and when you try to stop via supervisord it will not stop.

root     17614  0.0  0.3 227168 12632 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17723  6.5  0.4 235768 16684 ?        S    21:36   0:00 /bin/python /opt/cesi/cesi/web.py
nginx    17728  6.5  0.4 309592 16936 ?        Sl   21:36   0:00 /bin/python /opt/cesi/cesi/web.py
[root@supervisord cesi]# supervisorctl stop cesi
cesi: stopped
[root@supervisord cesi]# ps aux
.
.
root     17614  0.0  0.3 227320 12664 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17728  0.5  0.4 309592 16936 ?        Sl   21:36   0:01 /bin/python /opt/cesi/cesi/web.py
root     17738  0.0  0.0 123356  1380 pts/0    R+   21:40   0:00 ps aux
[root@supervisord cesi]# supervisorctl stop cesi
cesi: ERROR (not running)

As you see it’s not stopped, so simplest thing would be to change use_reloader in web.py

/opt/cesi/cesi/web.py
--- web.py.org	2014-12-27 21:23:37.625143414 +0200
+++ web.py	2014-12-27 21:23:48.222118215 +0200
@@ -531,7 +531,7 @@
 
 try:
     if __name__ == '__main__':
-        app.run(debug=True, use_reloader=True)
+        app.run(debug=True, use_reloader=False)
 except xmlrpclib.Fault as err:
     print "A fault occurred"
     print "Fault code: %d" % err.faultCode

Working like a charm..

[root@supervisord cesi]# supervisorctl start cesi
cesi: started
[root@supervisord cesi]# ps aux| grep cesi
nginx    17704  0.2  0.4 237048 18232 ?        S    21:30   0:00 /bin/python /opt/cesi/cesi/web.py

System side is ok and let us configure the nginx..

/etc/nginx/conf.d/default.conf
server {
        server_name localhost 192.168.9.240 212.213.214.215;
 
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:5000;
        }
 
        location /static {
                root /opt/cesi/cesi/;
        }
}

Let’s check TCP sockets

[root@supervisord cesi]# netstat -anp| grep LISTEN       
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      17761/python        
tcp        0      0 192.168.9.240:9001      0.0.0.0:*               LISTEN      17614/python        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      17842/nginx: master 
tcp        0      0 192.168.9.240:22        0.0.0.0:*               LISTEN      852/sshd

It’s time to login..


Login screen


Dashboard


All supervisors in one screen


A node


A process log

Kudos goes to Gülşah and Kaan and that’s all folks..

Best Way to Daemonize Applications on Linux

22nd December 2014 by Ali Erdinç Köroğlu

I tried to explain how to daemonize applications before but how about monitor and even start/stop/restart processes locally or remotely? Well, here is Supervisor. Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

As you know we need to write rc.d or systemd scripts for every single process instance. It’s hard to write and maintain also those scripts are not able to automatically restart a crashed process. So supervisord is the solution, it’s simple, efficient, centralized, extensible etc.. etc..

Supervisor has two component, supervisord and supervisorctl. Supervisord is the server side of supervisor and is responsible for starting child programs, controlling, logging and handling events. It’s also providing a web interface to view and control process status and an XML-RPC interface to control supervisor and the programs it runs. Supervisorctl on the otherhand, providing a shell-like interface for connecting to supervisord. But 1st let us install into our CentOS7 server. Supervisor is available on EPEL7 repository, if you don’t know how to add EPEL repository, please read this.

Installation is easy

[root@Neverland ~]# yum install supervisor

Please don’t forget to enable supervisor for systemd

[root@Neverland ~]# systemctl enable supervisord

It begins with a config file :)

/etc/supervisor.conf
[unix_http_server]
file=/var/tmp/supervisor.sock; (the path to the socket file)
 
[inet_http_server]      ; inet (TCP) server disabled by default 
port=192.168.1.1:9001   ; (ip_address:port specifier, *:port for all iface)
username=superv         ; (default is no username (open server))
password=superv         ; (default is no password (open server))
 
[supervisord]
logfile=/var/log/supervisor/supervisord.log  ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)
 
[rpcinterface:supervisor]
supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
 
[supervisorctl]
serverurl=unix:///var/tmp/supervisor.sock ; use a unix:// URL  for a unix socket
 
[include]
files = supervisord.d/*.ini

This process config file includes details such as directory, command, process name, process owner, logging etc. If you wanna know more, please read this.

/etc/supervisor.d/fixtures.ini
[program:fixtures]
directory=/opt/pronet/fixtures
command=/usr/java/jdk1.7.0_71/bin/java -Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties -Dlog4j..
process_name=%(program_name)s
user=pronet
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/pronet/fixtures-stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/var/log/pronet/fixtures-stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

So when you start or restart supervisord fixtures process will start or restart too.. (depends on your process config)

[root@Neverland ~]# systemctl start supervisord
[root@Neverland ~]# supervisorctl status fixtures
fixtures                         RUNNING    pid 5786, uptime 0:00:03

And you can monitor or control remotely..

[root@nagios ~]# supervisorctl -s http://192.168.1.1:9001 -u superv -p superv status fixtures
fixtures                         RUNNING    pid 5786, uptime 0:04:20
[root@nagios ~]# supervisorctl -s http://192.168.1.1:9001 -u superv -p superv restart fixtures
fixtures: stopped
fixtures: started

There are some centralized supervisord web interfaces but I’ll cover them later :)

I already explained how to daemonize java applications on SysV-style system in here. Since CentOS7/RHEL7 comes with Systemd which is a system and service manager for Linux we migrate old init scripts to the new system.

Again we’ll use our best budy Daemonize but this time we gonna compile it from source because although it’s signed as approved on fedora package db, Daemonize is not in EPEL7 repository for now. I’ll not going to details how to compile install etc. but I assume that you installed daemonize into /usr/local/sbin

So we need to create two files /etc/sysconfig/fixtures and /lib/systemd/system/fixtures.service
This 1st file is where we define java releated variables such as user, java path, arguments, log files etc..

/etc/sysconfig/fixtures
1
2
3
4
5
6
7
8
9
10
# Configz for fixtures service
 
JAVA_USER="pronet"
JAVA_STDOUT="/var/log/pronet/fixtures.log"
JAVA_STDERR="/var/log/pronet/fixtures-error.log"
JAVA_BIN="/usr/java/jdk1.7.0_71/bin/java"
JAVA_APPDIR="/opt/pronet/fixtures"
ARG1="-Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties"
ARG2="-Dlog4j.configuration=file:/opt/pronet/fixtures/fixtures-log.properties"
ARG3="-jar /opt/pronet/fixtures/fixtures.jar"

2nd file is service file for fixtures where we define systemd releted variables. There are plenty of documents in Freedesktop Systemd wiki, if you want to know more about I advice you to read them. But roughly unit: consist information about a service, a socket, a device etc, service: information about a process controlled and supervised by systemd and install: installation information for the unit

/lib/systemd/system/fixtures.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[Unit]
Description=Fixtures Service
After=syslog.target
After=network.target
 
[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/fixtures
ExecStart=/usr/local/sbin/daemonize -u $JAVA_USER -o $JAVA_STDOUT -e $JAVA_STDERR -c $JAVA_APPDIR $JAVA_BIN $ARG1 $ARG2 $ARG3
ExecStop=/bin/kill -TERM $MAINPID
TimeoutSec=300
 
[Install]
WantedBy=multi-user.target

Let’s start and stop the service

[root@Srv25 pronet]# systemctl start fixtures
[root@Srv25 pronet]# systemctl stop fixtures

if there is something wrong all service files and docker containers insert data into the systemd journal and we can read the journal :)

[root@Srv25 pronet]# journalctl -u fixtures.service

Checking the service status

[root@Srv25 pronet]# systemctl status fixtures
fixtures.service - Fixtures Service
   Loaded: loaded (/usr/lib/systemd/system/fixtures.service; disabled)
   Active: active (running) since Wed 2014-10-29 21:21:49 EET; 13min ago
 Main PID: 28859 (java)
   CGroup: /system.slice/fixtures.service
           └─28859 /usr/java/jdk1.7.0_71/bin/java -Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties -Dlog4j.configuration=file:...
 
Oct 29 21:21:49 Srv25 systemd[1]: Started Fixtures Service.

Enable the service to be started on bootup

[root@Srv25 pronet]# systemctl enable fixtures.service
ln -s '/usr/lib/systemd/system/fixtures.service' '/etc/systemd/system/multi-user.target.wants/fixtures.service'
[root@Srv25 pronet]#

So that’s how it works..

How to sync time properly: ntpdate or ntpd?

7th November 2013 by Ali Erdinç Köroğlu

Previously I explained how to install chrooted NTP server, but the question is how you’re going to sync time of your server with a NTP server. There are two options: Ntpdate and Ntpd.

Ntpdate is for the one-time synchronization only.
Ntpd (network time protocol daemon) is for automatically sync system time with a remote reference time server.

There are many examples like adding cronjobs for ntpdate hourly, daily, weekly etc. The main difference between ntpd and ntpdate; ntpd will run all the time and continuously adjust the system time when clocks drift but ntpdate will not. Also keep in mind that ntpdate is deprecated as of September 2012.

So why we need ntpdate at all ?
In ancient ages it was important to get the system time before starting ntpd and usually done by ntpdate. Over time, ntpd evolved and no longer necessary to set the time before starting ntpd.

To sum up; if you’re running time specific operations like application servers, database servers, email servers, clusters etc. ntpd is what you need.

Installation

Since NTP package is in the base repository no need to add extra repository.

yum install ntp
chkconfig ntpd on

Configuration

/etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
 
server 192.168.100.254          # your NTP server
server 0.tr.pool.ntp.org        # region releated ntp.org server
server ntp.ulakbim.gov.tr       # local authority

Since this will not be a ntp server for other so no need to listen on all interfaces.

/etc/sysconfig/ntpd
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g -I eth0"

Starting..

[root@cache ~]# /etc/init.d/ntpd start
Starting ntpd:                                             [  OK  ]

NTP query result and network time synchronisation status

[root@cache ~]# ntpstat 
synchronised to NTP server (192.168.100.254) at stratum 4 
   time correct to within 108 ms
   polling server every 64 s
[root@cache ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.100.254 82.94.167.75     3 u    5   64  377    0.276  -21.198  25.027

And as you see everything ok..

/var/log/messages
Nov  7 13:48:51 cache ntpd[44248]: ntpd 4.2.4p8@1.1612-o Fri Feb 22 11:23:27 UTC 2013 (1)
Nov  7 13:48:51 cache ntpd[44249]: precision = 0.079 usec
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #1 wildcard, ::#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #2 lo, ::1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #3 eth0, fe80::20c:29ff:febd:d65f#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #4 eth1, fe80::20c:29ff:febd:d669#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #5 lo, 127.0.0.1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #6 eth0, 192.168.100.1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #7 eth1, 192.168.101.1#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on routing socket on fd #29 for interface updates
Nov  7 13:48:51 cache ntpd[44249]: kernel time sync status 2040

Chrooted NTP Server on CentOS 6

27th July 2013 by Ali Erdinç Köroğlu

What is NTP? Network Time Protocol (NTP) is used to automatically sync system time with a remote reference time server. Why time synchronization is important? Because every aspect of managing, securing, planning, and debugging a network involves determining when events happen. Think about time-based AAA authentication and authorization, billing services, financial services, fault analysis.. Time management is a crucial component of healthy and secure network.

Why chroot? Security precaution :)

Scenario

We’ll create a ntp server for 2 different LAN (192.168.100 & 192.168.101) sync via pool.ntp.org and Turkish Academic Network and Information Center time servers.

Installation

Since NTP package is in the base repository there is no need to add extra repository.

yum install ntp
chkconfig ntpd on

Chroot Structure

There is no chroot-ntp package, we should prepare chroot environment.

mkdir /chroot
mkdir /chroot/ntp
mkdir /chroot/ntp/dev
mknod -m 666 /chroot/ntp/dev/null c 1 3
mknod -m 666 /chroot/ntp/dev/zero c 1 5
mknod -m 444 /chroot/ntp/dev/random c 1 8
mkdir /chroot/ntp/etc
mkdir /chroot/ntp/proc
mkdir /chroot/ntp/var
mkdir /chroot/ntp/var/lib
mkdir /chroot/ntp/var/lib/ntp
mv /var/lib/ntp/drift /chroot/ntp/var/lib/ntp/
chown -R ntp:ntp /chroot/ntp/var/lib/ntp
mkdir /chroot/ntp/var/log
mkdir /chroot/ntp/var/log/ntpstats
chown -R ntp:ntp /chroot/ntp/var/log/ntpstats
mv /etc/ntp.conf /chroot/ntp/etc
ln -s /chroot/ntp/etc/ntp.conf /etc/ntp.conf

Structure looks like this..

[root@firewall ~]# tree /chroot/ntp/
/chroot/ntp/
├── dev
│   ├── null
│   ├── random
│   └── zero
├── etc
│   └── ntp.conf
├── proc
└── var
    ├── lib
    │   └── ntp
    │       └── drift
    └── log
        └── ntpstats

Configuration

/chroot/ntp/etc/ntp.conf
server 0.tr.pool.ntp.org
server ntp.ulakbim.gov.tr
server 127.127.1.0
fudge 127.127.1.0 stratum 10
 
restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.101.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
 
driftfile /var/lib/ntp/drift
logfile /var/log/ntp.log
/etc/sysconfig/ntpd
OPTIONS="-i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g"

NTP requires proc file system in chroot environment, you could mount manually but I modified ntpd initscript.

diff -u /etc/init.d/ntpd.org /etc/init.d/ntpd
--- /etc/init.d/ntpd.org	2013-07-22 18:33:23.553385624 +0300
+++ /etc/init.d/ntpd	2013-07-24 11:22:47.594735735 +0300
@@ -30,6 +30,27 @@
 
 prog=ntpd
 lockfile=/var/lock/subsys/$prog
+chroot=/chroot/ntp
+
+mount_proc() {
+        echo -n $"Binding proc to chroot environment: "
+        ret=0
+        mount --bind /proc $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
+
+umount_proc (){
+        echo -n $"Unmounting proc from chroot environment: "
+        ret=0
+       umount $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
 
 start() {
        [ "$EUID" != "0" ] && exit 4
@@ -38,6 +59,9 @@
        [ -f /etc/sysconfig/ntpd ] || exit 6
        . /etc/sysconfig/ntpd
 
+       # Mounting proc into chroot
+       mount_proc
+
         # Start daemons.
         echo -n $"Starting $prog: "
         daemon $prog $OPTIONS
@@ -54,6 +78,10 @@
        RETVAL=$?
         echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
+
+       #Unmount proc from chroot
+       umount_proc
+
        return $RETVAL
 }

Let’s start the server..

[root@firewall ntp]# /etc/init.d/ntpd start
Binding proc to chroot environment:                        [  OK  ]
Starting ntpd:                                             [  OK  ]

Just to make sure everything is ok or not :)

[root@firewall ntpstats]# ps aux | grep ntpd
root     23824  0.0  0.0 103236   852 pts/0    S+   13:15   0:00 grep ntpd
ntp      25301  0.0  0.0  30164  1628 ?        Ss   Jul24   0:01 ntpd -i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g[root@firewall ntpstats]# cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1953976k,nr_inodes=488494,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda1 / ext4 rw,noatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
proc /chroot/ntp/proc proc rw,relatime 0 0

NTP query result and network time synchronisation status

[root@firewall ntp]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*195.50.171.101  145.253.2.212    2 u  420 1024  377   69.880   -0.031   0.006
+samur.ulak.net. 131.188.3.221    2 u  352 1024  377   30.842   -2.137   3.257
 LOCAL(0)        .LOCL.          10 l   27   64  377    0.000    0.000   0.000
[root@firewall ntp]# ntpstat 
synchronised to NTP server (195.50.171.101) at stratum 3 
   time correct to within 84 ms
   polling server every 1024 s

Chrooted Master and Slave DNS Configuration on CentOS 6

16th July 2013 by Ali Erdinç Köroğlu

As I already explained in Chrooted PHP-FPM with Nginx on CentOS 6, chroot is about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. Why chroot in a DNS Server? Since DNS is within the top network security attack targets, chroot considered as a security precaution to limit the amount of access any malicious individual could gain by exploiting vulnerabilities in BIND (the Berkeley Internet Name Domain).

Scenario

We’ll create two chrooted authoritative (non-recursive) only name server as ns1.koroglu.org (1.1.1.1) and ns2.koroglu.org (2.2.2.2). ns1 will be configured as master and ns2 as slave dns server, taking its zone data from ns1.

Installation

Let’s begin, bytheway please don’t forget to do this installation on both server..

[root@8bitplus-dns ~]# yum install bind-chroot bind bind-utils bind-libs

Chroot environment comes with bind-chroot RPM, so you dont have to figure out the structure :)

[root@8bitplus-dns ~]# rpm -ql bind-chroot
/var/named/chroot
/var/named/chroot/dev
/var/named/chroot/dev/null
/var/named/chroot/dev/random
/var/named/chroot/dev/zero
/var/named/chroot/etc
/var/named/chroot/etc/localtime
/var/named/chroot/etc/named
/var/named/chroot/etc/named.conf
/var/named/chroot/etc/pki
/var/named/chroot/etc/pki/dnssec-keys
/var/named/chroot/usr
/var/named/chroot/usr/lib64
/var/named/chroot/usr/lib64/bind
/var/named/chroot/var
/var/named/chroot/var/log
/var/named/chroot/var/named
/var/named/chroot/var/run
/var/named/chroot/var/run/named
/var/named/chroot/var/tmp

Configuration

Copy related config files into chroot environment on both server..

[root@8bitplus-dns ~]# cp /usr/share/doc/bind-9.8.2/sample/var/named/named.* /var/named/chroot/var/named/
[root@8bitplus-dns ~]# mkdir /var/named/chroot/var/named/data
[root@8bitplus-dns ~]# chown -R named:named /var/named/chroot/var/named/

Master Name Server

named.conf is the configuration file for BIND DNS server.

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// Koroglu.org Master DNS Configuration
 
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        dnssec-enable yes;
        notify yes;
        also-notify { 2.2.2.2; };
        allow-transfer { 2.2.2.2; };
};
 
controls { };
 
logging {
        channel master_log {
                file "/var/log/named/master.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                master_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type master;
        file "koroglu.org";
};

And the zone file.. It’s a text file that describes a portion of a domain name space using the Domain Name System. The format of a zone file is defined in RFC 1035 section 5 and RFC 1034 section 3.6. We’ll define our DNS servers and four subdomain; ns1, ns2, www and ae

/var/named/chroot/var/named/koroglu.org
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$TTL 86400
@               IN      SOA     ns1.koroglu.org. erdinc.8bitplus.com. (
                              2012030703 ; serial
                              43200      ; refresh (12hours)
                              3600       ; retry (1hour)
                              1209600    ; expire (2week)
                              6200)      ; TTL minimum (2hour)
 
                        IN      NS      ns1.koroglu.org.
                        IN      NS      ns2.koroglu.org.
                        IN      A       2.2.2.2
 
ns1                     IN      A       1.1.1.1
ns2                     IN      A       2.2.2.2
www                     IN      A       2.2.2.2
ae                      IN      CNAME   www.koroglu.org.

Slave Name Server

named.conf configuration for slave DNS

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Koroglu.org Slave DNS Configuration
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        dump-file "/var/named/data/cache_dump.db";
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-notify { 1.1.1.1; };
};
 
controls { };
 
logging {
        channel slave_log {
                file "/var/log/named/slave.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                slave_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type slave;
        file "slaves/koroglu.org";
        masters { 1.1.1.1; };
};

No zone file created manually, because slave dns server will be taking its zone data from master.

Advices

  1. At least two DNS server should be configured
  2. Master and slave servers should not be on the same subnet, may cause single point of failure
  3. Be carefull about the file permissions in chroot environment
  4. Always check named log file in case of any problem
  5. I didn’t apply RNDC on both server cause I would like to minimize the risk

Chrooted PHP-FPM with Nginx on CentOS 6

11th July 2013 by Ali Erdinç Köroğlu

What is chroot (change root)? It’s about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. When you change root to another directory you can not access files and commands outside that directory. As you see, chroot enhances the security for the system and creates a virtual environment inside of which the application operates. If a vulnerability exist in the application or code such that an attacker can gain file system access, who would only be able to access files inside virtualized environment and the rest of the operating system and directory structure would remain inaccessable.

It’s good but chroot != security

Installation

Before installation, we should add Nginx repository to /etc/yum/repos.d. EPEL repository has Nginx too but the version is very old.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Installing..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configurations

/etc/php-fpm.d/example.conf
[example]
listen = /tmp/example.sock 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/example-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
chroot = /chroot/jailchdir = / 
php_admin_value[disable_functions] = dir,chdir,opendir,readdirphp_admin_value[error_log] = /var/log/php-fpm/example-error.log
php_admin_flag[log_errors] = on
php_admin_value[error_reporting] = E_ALL & ~E_NOTICE
 
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
/etc/nginx/conf.d/example.conf
server {
        server_name ae.koroglu.org;
        root /srv-www/example;        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/example.access.log main;
        error_log /var/log/nginx/example.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ $uri.php;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/example.sock;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;        }
}

Chroot Debugging

Beginning was simple, a classic PHP configuration error..

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings.
You are required to use the date.timezone setting or the date_default_timezone_set() function.
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier.
/etc/php.ini
[Date]
date.timezone = Europe/Istanbul

PHP requires date.timezone information so I added “Europe/Istanbul” into php.ini and copied related files into chroot environment. Rest requires tracing.. 1st step is to find example pool php-fpm processes..

[root@8bitplus ~]# ps aux | grep php | grep example
nginx    11947  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11949  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11950  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11951  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11952  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example

The minimum number of child processes can not be less than 5 for any PHP-FPM pool by default so we should trace related pool process ids.

[root@8bitplus ~]# strace -p 11947 -o chroot1.txt&
[1] 12650
[root@8bitplus ~]# Process 11947 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11949 -o chroot2.txt&
[2] 12651
[root@8bitplus ~]# Process 11949 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11950 -o chroot3.txt&
[3] 12652
[root@8bitplus ~]# Process 11950 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11951 -o chroot4.txt&
[4] 12653
[root@8bitplus ~]# Process 11951 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11952 -o chroot5.txt&
[5] 12654
[root@8bitplus ~]# Process 11952 attached - interrupt to quit

I checked strace outputs it seems that /etc/resolv.conf, /etc/hosts, /etc/host.conf, /lib64/libnss_dns-2.12.so and /usr/lib64/libsoftokn3.so files needed for PHP-FPM chroot environment. Also you should know more about those files like where they come from, dependencies etc.. This will be very important while updating the system. If any upgrade operation updates those packages, your chroot may not be working due to broken shared library dependencies.

[root@8bitplus ~]# rpm -qf /lib64/libnss_dns-2.12.so /usr/lib64/libsoftokn3.so /usr/share/zoneinfo/Europe/Istanbul 
glibc-2.12-1.107.el6.x86_64nss-softokn-3.12.9-11.el6.x86_64tzdata-2013c-2.el6.noarch

Since PHP-FPM is not in the same chroot environment with Nginx, you should either make sure that both PHP-FPM and Nginx using the same chroot (Nginx should be chrooted too) or you should create soft link from Nginx document root folder to chrooted document root folder and that’s what I did..

[root@8bitplus srv-www]# pwd
/srv-www[root@8bitplus srv-www]# ls -l
total 0
lrwxrwxrwx 1 root root 34 Jul  2 18:30 example -> /chroot/jail/srv-www/example/

We could create multiple mysqld processes that listen for connections on different Unix socket files and TCP/IP ports but practical and easy way is to use TCP socket instead of dealing with mysqld_multi configuration.

/etc/my.cnf
#skip-networking
bind-address = 127.0.0.1

Also create urandom device for chroot

[root@8bitplus srv-www]# mknod -m 644 /chroot/jail/dev/urandom c 1 9

Chroot Structure

[root@8bitplus /]# tree chroot
chroot/
└── jail
    ├── dev
    │   └── urandom
    ├── etc
    │   ├── host.conf
    │   ├── hosts
    │   ├── localtime
    │   └── resolv.conf
    ├── lib64
    │   ├── libnss_dns-2.12.so
    │   └── libnss_dns.so.2 -> libnss_dns-2.12.so
    ├── srv-www
    │   └── example
    ├── usr
    │   ├── lib64
    │   │   └── libsoftokn3.so
    │   └── share
    │       └── zoneinfo
    │           ├── Europe
    │           │   └── Istanbul
    │           └── zone.tab
    └── var
        ├── lib
        │   └── php
        │       └── session
        └── log
            └── php-fpm

In this example you can use single chroot environment with multiple PHP-FPM pool. But if you’re using WordPress or any application which requires opendir function, I would strongly recommended that to create different chroot environment for each. You could test your chroot environment with PhpSpy Shell

« Previous Entries