The point of no return

Chrooted NTP Server on CentOS 6

27th July 2013 by Ali Erdinç Köroğlu

What is NTP? Network Time Protocol (NTP) is used to automatically sync system time with a remote reference time server. Why time synchronization is important? Because every aspect of managing, securing, planning, and debugging a network involves determining when events happen. Think about time-based AAA authentication and authorization, billing services, financial services, fault analysis.. Time management is a crucial component of healthy and secure network.

Why chroot? Security precaution :)

Scenario

We’ll create a ntp server for 2 different LAN (192.168.100 & 192.168.101) sync via pool.ntp.org and Turkish Academic Network and Information Center time servers.

Installation

Since NTP package is in the base repository there is no need to add extra repository.

yum install ntp
chkconfig ntpd on

Chroot Structure

There is no chroot-ntp package, we should prepare chroot environment.

mkdir /chroot
mkdir /chroot/ntp
mkdir /chroot/ntp/dev
mknod -m 666 /chroot/ntp/dev/null c 1 3
mknod -m 666 /chroot/ntp/dev/zero c 1 5
mknod -m 444 /chroot/ntp/dev/random c 1 8
mkdir /chroot/ntp/etc
mkdir /chroot/ntp/proc
mkdir /chroot/ntp/var
mkdir /chroot/ntp/var/lib
mkdir /chroot/ntp/var/lib/ntp
mv /var/lib/ntp/drift /chroot/ntp/var/lib/ntp/
chown -R ntp:ntp /chroot/ntp/var/lib/ntp
mkdir /chroot/ntp/var/log
mkdir /chroot/ntp/var/log/ntpstats
chown -R ntp:ntp /chroot/ntp/var/log/ntpstats
mv /etc/ntp.conf /chroot/ntp/etc
ln -s /chroot/ntp/etc/ntp.conf /etc/ntp.conf

Structure looks like this..

[root@firewall ~]# tree /chroot/ntp/
/chroot/ntp/
├── dev
│   ├── null
│   ├── random
│   └── zero
├── etc
│   └── ntp.conf
├── proc
└── var
    ├── lib
    │   └── ntp
    │       └── drift
    └── log
        └── ntpstats

Configuration

/chroot/ntp/etc/ntp.conf
server 0.tr.pool.ntp.org
server ntp.ulakbim.gov.tr
server 127.127.1.0
fudge 127.127.1.0 stratum 10
 
restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.101.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
 
driftfile /var/lib/ntp/drift
logfile /var/log/ntp.log
/etc/sysconfig/ntpd
OPTIONS="-i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g"

NTP requires proc file system in chroot environment, you could mount manually but I modified ntpd initscript.

diff -u /etc/init.d/ntpd.org /etc/init.d/ntpd
--- /etc/init.d/ntpd.org	2013-07-22 18:33:23.553385624 +0300
+++ /etc/init.d/ntpd	2013-07-24 11:22:47.594735735 +0300
@@ -30,6 +30,27 @@
 
 prog=ntpd
 lockfile=/var/lock/subsys/$prog
+chroot=/chroot/ntp
+
+mount_proc() {
+        echo -n $"Binding proc to chroot environment: "
+        ret=0
+        mount --bind /proc $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
+
+umount_proc (){
+        echo -n $"Unmounting proc from chroot environment: "
+        ret=0
+       umount $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
 
 start() {
        [ "$EUID" != "0" ] && exit 4
@@ -38,6 +59,9 @@
        [ -f /etc/sysconfig/ntpd ] || exit 6
        . /etc/sysconfig/ntpd
 
+       # Mounting proc into chroot
+       mount_proc
+
         # Start daemons.
         echo -n $"Starting $prog: "
         daemon $prog $OPTIONS
@@ -54,6 +78,10 @@
        RETVAL=$?
         echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
+
+       #Unmount proc from chroot
+       umount_proc
+
        return $RETVAL
 }

Let’s start the server..

[root@firewall ntp]# /etc/init.d/ntpd start
Binding proc to chroot environment:                        [  OK  ]
Starting ntpd:                                             [  OK  ]

Just to make sure everything is ok or not :)

[root@firewall ntpstats]# ps aux | grep ntpd
root     23824  0.0  0.0 103236   852 pts/0    S+   13:15   0:00 grep ntpd
ntp      25301  0.0  0.0  30164  1628 ?        Ss   Jul24   0:01 ntpd -i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g[root@firewall ntpstats]# cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1953976k,nr_inodes=488494,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda1 / ext4 rw,noatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
proc /chroot/ntp/proc proc rw,relatime 0 0

NTP query result and network time synchronisation status

[root@firewall ntp]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*195.50.171.101  145.253.2.212    2 u  420 1024  377   69.880   -0.031   0.006
+samur.ulak.net. 131.188.3.221    2 u  352 1024  377   30.842   -2.137   3.257
 LOCAL(0)        .LOCL.          10 l   27   64  377    0.000    0.000   0.000
[root@firewall ntp]# ntpstat 
synchronised to NTP server (195.50.171.101) at stratum 3 
   time correct to within 84 ms
   polling server every 1024 s

Chrooted Master and Slave DNS Configuration on CentOS 6

16th July 2013 by Ali Erdinç Köroğlu

As I already explained in Chrooted PHP-FPM with Nginx on CentOS 6, chroot is about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. Why chroot in a DNS Server? Since DNS is within the top network security attack targets, chroot considered as a security precaution to limit the amount of access any malicious individual could gain by exploiting vulnerabilities in BIND (the Berkeley Internet Name Domain).

Scenario

We’ll create two chrooted authoritative (non-recursive) only name server as ns1.koroglu.org (1.1.1.1) and ns2.koroglu.org (2.2.2.2). ns1 will be configured as master and ns2 as slave dns server, taking its zone data from ns1.

Installation

Let’s begin, bytheway please don’t forget to do this installation on both server..

[root@8bitplus-dns ~]# yum install bind-chroot bind bind-utils bind-libs

Chroot environment comes with bind-chroot RPM, so you dont have to figure out the structure :)

[root@8bitplus-dns ~]# rpm -ql bind-chroot
/var/named/chroot
/var/named/chroot/dev
/var/named/chroot/dev/null
/var/named/chroot/dev/random
/var/named/chroot/dev/zero
/var/named/chroot/etc
/var/named/chroot/etc/localtime
/var/named/chroot/etc/named
/var/named/chroot/etc/named.conf
/var/named/chroot/etc/pki
/var/named/chroot/etc/pki/dnssec-keys
/var/named/chroot/usr
/var/named/chroot/usr/lib64
/var/named/chroot/usr/lib64/bind
/var/named/chroot/var
/var/named/chroot/var/log
/var/named/chroot/var/named
/var/named/chroot/var/run
/var/named/chroot/var/run/named
/var/named/chroot/var/tmp

Configuration

Copy related config files into chroot environment on both server..

[root@8bitplus-dns ~]# cp /usr/share/doc/bind-9.8.2/sample/var/named/named.* /var/named/chroot/var/named/
[root@8bitplus-dns ~]# mkdir /var/named/chroot/var/named/data
[root@8bitplus-dns ~]# chown -R named:named /var/named/chroot/var/named/

Master Name Server

named.conf is the configuration file for BIND DNS server.

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// Koroglu.org Master DNS Configuration
 
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        dnssec-enable yes;
        notify yes;
        also-notify { 2.2.2.2; };
        allow-transfer { 2.2.2.2; };
};
 
controls { };
 
logging {
        channel master_log {
                file "/var/log/named/master.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                master_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type master;
        file "koroglu.org";
};

And the zone file.. It’s a text file that describes a portion of a domain name space using the Domain Name System. The format of a zone file is defined in RFC 1035 section 5 and RFC 1034 section 3.6. We’ll define our DNS servers and four subdomain; ns1, ns2, www and ae

/var/named/chroot/var/named/koroglu.org
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$TTL 86400
@               IN      SOA     ns1.koroglu.org. erdinc.8bitplus.com. (
                              2012030703 ; serial
                              43200      ; refresh (12hours)
                              3600       ; retry (1hour)
                              1209600    ; expire (2week)
                              6200)      ; TTL minimum (2hour)
 
                        IN      NS      ns1.koroglu.org.
                        IN      NS      ns2.koroglu.org.
                        IN      A       2.2.2.2
 
ns1                     IN      A       1.1.1.1
ns2                     IN      A       2.2.2.2
www                     IN      A       2.2.2.2
ae                      IN      CNAME   www.koroglu.org.

Slave Name Server

named.conf configuration for slave DNS

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Koroglu.org Slave DNS Configuration
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        dump-file "/var/named/data/cache_dump.db";
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-notify { 1.1.1.1; };
};
 
controls { };
 
logging {
        channel slave_log {
                file "/var/log/named/slave.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                slave_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type slave;
        file "slaves/koroglu.org";
        masters { 1.1.1.1; };
};

No zone file created manually, because slave dns server will be taking its zone data from master.

Advices

  1. At least two DNS server should be configured
  2. Master and slave servers should not be on the same subnet, may cause single point of failure
  3. Be carefull about the file permissions in chroot environment
  4. Always check named log file in case of any problem
  5. I didn’t apply RNDC on both server cause I would like to minimize the risk

Chrooted PHP-FPM with Nginx on CentOS 6

11th July 2013 by Ali Erdinç Köroğlu

What is chroot (change root)? It’s about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. When you change root to another directory you can not access files and commands outside that directory. As you see, chroot enhances the security for the system and creates a virtual environment inside of which the application operates. If a vulnerability exist in the application or code such that an attacker can gain file system access, who would only be able to access files inside virtualized environment and the rest of the operating system and directory structure would remain inaccessable.

It’s good but chroot != security

Installation

Before installation, we should add Nginx repository to /etc/yum/repos.d. EPEL repository has Nginx too but the version is very old.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Installing..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configurations

/etc/php-fpm.d/example.conf
[example]
listen = /tmp/example.sock 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/example-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
chroot = /chroot/jailchdir = / 
php_admin_value[disable_functions] = dir,chdir,opendir,readdirphp_admin_value[error_log] = /var/log/php-fpm/example-error.log
php_admin_flag[log_errors] = on
php_admin_value[error_reporting] = E_ALL & ~E_NOTICE
 
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
/etc/nginx/conf.d/example.conf
server {
        server_name ae.koroglu.org;
        root /srv-www/example;        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/example.access.log main;
        error_log /var/log/nginx/example.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ $uri.php;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/example.sock;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;        }
}

Chroot Debugging

Beginning was simple, a classic PHP configuration error..

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings.
You are required to use the date.timezone setting or the date_default_timezone_set() function.
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier.
/etc/php.ini
[Date]
date.timezone = Europe/Istanbul

PHP requires date.timezone information so I added “Europe/Istanbul” into php.ini and copied related files into chroot environment. Rest requires tracing.. 1st step is to find example pool php-fpm processes..

[root@8bitplus ~]# ps aux | grep php | grep example
nginx    11947  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11949  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11950  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11951  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11952  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example

The minimum number of child processes can not be less than 5 for any PHP-FPM pool by default so we should trace related pool process ids.

[root@8bitplus ~]# strace -p 11947 -o chroot1.txt&
[1] 12650
[root@8bitplus ~]# Process 11947 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11949 -o chroot2.txt&
[2] 12651
[root@8bitplus ~]# Process 11949 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11950 -o chroot3.txt&
[3] 12652
[root@8bitplus ~]# Process 11950 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11951 -o chroot4.txt&
[4] 12653
[root@8bitplus ~]# Process 11951 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11952 -o chroot5.txt&
[5] 12654
[root@8bitplus ~]# Process 11952 attached - interrupt to quit

I checked strace outputs it seems that /etc/resolv.conf, /etc/hosts, /etc/host.conf, /lib64/libnss_dns-2.12.so and /usr/lib64/libsoftokn3.so files needed for PHP-FPM chroot environment. Also you should know more about those files like where they come from, dependencies etc.. This will be very important while updating the system. If any upgrade operation updates those packages, your chroot may not be working due to broken shared library dependencies.

[root@8bitplus ~]# rpm -qf /lib64/libnss_dns-2.12.so /usr/lib64/libsoftokn3.so /usr/share/zoneinfo/Europe/Istanbul 
glibc-2.12-1.107.el6.x86_64nss-softokn-3.12.9-11.el6.x86_64tzdata-2013c-2.el6.noarch

Since PHP-FPM is not in the same chroot environment with Nginx, you should either make sure that both PHP-FPM and Nginx using the same chroot (Nginx should be chrooted too) or you should create soft link from Nginx document root folder to chrooted document root folder and that’s what I did..

[root@8bitplus srv-www]# pwd
/srv-www[root@8bitplus srv-www]# ls -l
total 0
lrwxrwxrwx 1 root root 34 Jul  2 18:30 example -> /chroot/jail/srv-www/example/

We could create multiple mysqld processes that listen for connections on different Unix socket files and TCP/IP ports but practical and easy way is to use TCP socket instead of dealing with mysqld_multi configuration.

/etc/my.cnf
#skip-networking
bind-address = 127.0.0.1

Also create urandom device for chroot

[root@8bitplus srv-www]# mknod -m 644 /chroot/jail/dev/urandom c 1 9

Chroot Structure

[root@8bitplus /]# tree chroot
chroot/
└── jail
    ├── dev
    │   └── urandom
    ├── etc
    │   ├── host.conf
    │   ├── hosts
    │   ├── localtime
    │   └── resolv.conf
    ├── lib64
    │   ├── libnss_dns-2.12.so
    │   └── libnss_dns.so.2 -> libnss_dns-2.12.so
    ├── srv-www
    │   └── example
    ├── usr
    │   ├── lib64
    │   │   └── libsoftokn3.so
    │   └── share
    │       └── zoneinfo
    │           ├── Europe
    │           │   └── Istanbul
    │           └── zone.tab
    └── var
        ├── lib
        │   └── php
        │       └── session
        └── log
            └── php-fpm

In this example you can use single chroot environment with multiple PHP-FPM pool. But if you’re using WordPress or any application which requires opendir function, I would strongly recommended that to create different chroot environment for each. You could test your chroot environment with PhpSpy Shell