The point of no return

How to sync time properly: ntpdate or ntpd?

7th November 2013 by Ali Erdinç Köroğlu

Previously I explained how to install chrooted NTP server, but the question is how you’re going to sync time of your server with a NTP server. There are two options: Ntpdate and Ntpd.

Ntpdate is for the one-time synchronization only.
Ntpd (network time protocol daemon) is for automatically sync system time with a remote reference time server.

There are many examples like adding cronjobs for ntpdate hourly, daily, weekly etc. The main difference between ntpd and ntpdate; ntpd will run all the time and continuously adjust the system time when clocks drift but ntpdate will not. Also keep in mind that ntpdate is deprecated as of September 2012.

So why we need ntpdate at all ?
In ancient ages it was important to get the system time before starting ntpd and usually done by ntpdate. Over time, ntpd evolved and no longer necessary to set the time before starting ntpd.

To sum up; if you’re running time specific operations like application servers, database servers, email servers, clusters etc. ntpd is what you need.

Installation

Since NTP package is in the base repository no need to add extra repository.

yum install ntp
chkconfig ntpd on

Configuration

/etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
 
server 192.168.100.254          # your NTP server
server 0.tr.pool.ntp.org        # region releated ntp.org server
server ntp.ulakbim.gov.tr       # local authority

Since this will not be a ntp server for other so no need to listen on all interfaces.

/etc/sysconfig/ntpd
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g -I eth0"

Starting..

[root@cache ~]# /etc/init.d/ntpd start
Starting ntpd:                                             [  OK  ]

NTP query result and network time synchronisation status

[root@cache ~]# ntpstat 
synchronised to NTP server (192.168.100.254) at stratum 4 
   time correct to within 108 ms
   polling server every 64 s
[root@cache ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.100.254 82.94.167.75     3 u    5   64  377    0.276  -21.198  25.027

And as you see everything ok..

/var/log/messages
Nov  7 13:48:51 cache ntpd[44248]: ntpd 4.2.4p8@1.1612-o Fri Feb 22 11:23:27 UTC 2013 (1)
Nov  7 13:48:51 cache ntpd[44249]: precision = 0.079 usec
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #1 wildcard, ::#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #2 lo, ::1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #3 eth0, fe80::20c:29ff:febd:d65f#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #4 eth1, fe80::20c:29ff:febd:d669#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on interface #5 lo, 127.0.0.1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #6 eth0, 192.168.100.1#123 EnabledNov  7 13:48:51 cache ntpd[44249]: Listening on interface #7 eth1, 192.168.101.1#123 Disabled
Nov  7 13:48:51 cache ntpd[44249]: Listening on routing socket on fd #29 for interface updates
Nov  7 13:48:51 cache ntpd[44249]: kernel time sync status 2040

Chrooted NTP Server on CentOS 6

27th July 2013 by Ali Erdinç Köroğlu

What is NTP? Network Time Protocol (NTP) is used to automatically sync system time with a remote reference time server. Why time synchronization is important? Because every aspect of managing, securing, planning, and debugging a network involves determining when events happen. Think about time-based AAA authentication and authorization, billing services, financial services, fault analysis.. Time management is a crucial component of healthy and secure network.

Why chroot? Security precaution :)

Scenario

We’ll create a ntp server for 2 different LAN (192.168.100 & 192.168.101) sync via pool.ntp.org and Turkish Academic Network and Information Center time servers.

Installation

Since NTP package is in the base repository there is no need to add extra repository.

yum install ntp
chkconfig ntpd on

Chroot Structure

There is no chroot-ntp package, we should prepare chroot environment.

mkdir /chroot
mkdir /chroot/ntp
mkdir /chroot/ntp/dev
mknod -m 666 /chroot/ntp/dev/null c 1 3
mknod -m 666 /chroot/ntp/dev/zero c 1 5
mknod -m 444 /chroot/ntp/dev/random c 1 8
mkdir /chroot/ntp/etc
mkdir /chroot/ntp/proc
mkdir /chroot/ntp/var
mkdir /chroot/ntp/var/lib
mkdir /chroot/ntp/var/lib/ntp
mv /var/lib/ntp/drift /chroot/ntp/var/lib/ntp/
chown -R ntp:ntp /chroot/ntp/var/lib/ntp
mkdir /chroot/ntp/var/log
mkdir /chroot/ntp/var/log/ntpstats
chown -R ntp:ntp /chroot/ntp/var/log/ntpstats
mv /etc/ntp.conf /chroot/ntp/etc
ln -s /chroot/ntp/etc/ntp.conf /etc/ntp.conf

Structure looks like this..

[root@firewall ~]# tree /chroot/ntp/
/chroot/ntp/
├── dev
│   ├── null
│   ├── random
│   └── zero
├── etc
│   └── ntp.conf
├── proc
└── var
    ├── lib
    │   └── ntp
    │       └── drift
    └── log
        └── ntpstats

Configuration

/chroot/ntp/etc/ntp.conf
server 0.tr.pool.ntp.org
server ntp.ulakbim.gov.tr
server 127.127.1.0
fudge 127.127.1.0 stratum 10
 
restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.101.0 mask 255.255.255.0 nomodify notrap
restrict 127.0.0.1
 
driftfile /var/lib/ntp/drift
logfile /var/log/ntp.log
/etc/sysconfig/ntpd
OPTIONS="-i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g"

NTP requires proc file system in chroot environment, you could mount manually but I modified ntpd initscript.

diff -u /etc/init.d/ntpd.org /etc/init.d/ntpd
--- /etc/init.d/ntpd.org	2013-07-22 18:33:23.553385624 +0300
+++ /etc/init.d/ntpd	2013-07-24 11:22:47.594735735 +0300
@@ -30,6 +30,27 @@
 
 prog=ntpd
 lockfile=/var/lock/subsys/$prog
+chroot=/chroot/ntp
+
+mount_proc() {
+        echo -n $"Binding proc to chroot environment: "
+        ret=0
+        mount --bind /proc $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
+
+umount_proc (){
+        echo -n $"Unmounting proc from chroot environment: "
+        ret=0
+       umount $chroot/proc
+        let ret+=$?;
+        [ $ret -eq 0 ] && success || failure
+        echo
+        return $ret
+}
 
 start() {
        [ "$EUID" != "0" ] && exit 4
@@ -38,6 +59,9 @@
        [ -f /etc/sysconfig/ntpd ] || exit 6
        . /etc/sysconfig/ntpd
 
+       # Mounting proc into chroot
+       mount_proc
+
         # Start daemons.
         echo -n $"Starting $prog: "
         daemon $prog $OPTIONS
@@ -54,6 +78,10 @@
        RETVAL=$?
         echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
+
+       #Unmount proc from chroot
+       umount_proc
+
        return $RETVAL
 }

Let’s start the server..

[root@firewall ntp]# /etc/init.d/ntpd start
Binding proc to chroot environment:                        [  OK  ]
Starting ntpd:                                             [  OK  ]

Just to make sure everything is ok or not :)

[root@firewall ntpstats]# ps aux | grep ntpd
root     23824  0.0  0.0 103236   852 pts/0    S+   13:15   0:00 grep ntpd
ntp      25301  0.0  0.0  30164  1628 ?        Ss   Jul24   0:01 ntpd -i /chroot/ntp -u ntp:ntp -p /var/run/ntpd.pid -g[root@firewall ntpstats]# cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1953976k,nr_inodes=488494,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda1 / ext4 rw,noatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
proc /chroot/ntp/proc proc rw,relatime 0 0

NTP query result and network time synchronisation status

[root@firewall ntp]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*195.50.171.101  145.253.2.212    2 u  420 1024  377   69.880   -0.031   0.006
+samur.ulak.net. 131.188.3.221    2 u  352 1024  377   30.842   -2.137   3.257
 LOCAL(0)        .LOCL.          10 l   27   64  377    0.000    0.000   0.000
[root@firewall ntp]# ntpstat 
synchronised to NTP server (195.50.171.101) at stratum 3 
   time correct to within 84 ms
   polling server every 1024 s

Chrooted Master and Slave DNS Configuration on CentOS 6

16th July 2013 by Ali Erdinç Köroğlu

As I already explained in Chrooted PHP-FPM with Nginx on CentOS 6, chroot is about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. Why chroot in a DNS Server? Since DNS is within the top network security attack targets, chroot considered as a security precaution to limit the amount of access any malicious individual could gain by exploiting vulnerabilities in BIND (the Berkeley Internet Name Domain).

Scenario

We’ll create two chrooted authoritative (non-recursive) only name server as ns1.koroglu.org (1.1.1.1) and ns2.koroglu.org (2.2.2.2). ns1 will be configured as master and ns2 as slave dns server, taking its zone data from ns1.

Installation

Let’s begin, bytheway please don’t forget to do this installation on both server..

[root@8bitplus-dns ~]# yum install bind-chroot bind bind-utils bind-libs

Chroot environment comes with bind-chroot RPM, so you dont have to figure out the structure :)

[root@8bitplus-dns ~]# rpm -ql bind-chroot
/var/named/chroot
/var/named/chroot/dev
/var/named/chroot/dev/null
/var/named/chroot/dev/random
/var/named/chroot/dev/zero
/var/named/chroot/etc
/var/named/chroot/etc/localtime
/var/named/chroot/etc/named
/var/named/chroot/etc/named.conf
/var/named/chroot/etc/pki
/var/named/chroot/etc/pki/dnssec-keys
/var/named/chroot/usr
/var/named/chroot/usr/lib64
/var/named/chroot/usr/lib64/bind
/var/named/chroot/var
/var/named/chroot/var/log
/var/named/chroot/var/named
/var/named/chroot/var/run
/var/named/chroot/var/run/named
/var/named/chroot/var/tmp

Configuration

Copy related config files into chroot environment on both server..

[root@8bitplus-dns ~]# cp /usr/share/doc/bind-9.8.2/sample/var/named/named.* /var/named/chroot/var/named/
[root@8bitplus-dns ~]# mkdir /var/named/chroot/var/named/data
[root@8bitplus-dns ~]# chown -R named:named /var/named/chroot/var/named/

Master Name Server

named.conf is the configuration file for BIND DNS server.

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
// Koroglu.org Master DNS Configuration
 
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        dnssec-enable yes;
        notify yes;
        also-notify { 2.2.2.2; };
        allow-transfer { 2.2.2.2; };
};
 
controls { };
 
logging {
        channel master_log {
                file "/var/log/named/master.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                master_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type master;
        file "koroglu.org";
};

And the zone file.. It’s a text file that describes a portion of a domain name space using the Domain Name System. The format of a zone file is defined in RFC 1035 section 5 and RFC 1034 section 3.6. We’ll define our DNS servers and four subdomain; ns1, ns2, www and ae

/var/named/chroot/var/named/koroglu.org
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$TTL 86400
@               IN      SOA     ns1.koroglu.org. erdinc.8bitplus.com. (
                              2012030703 ; serial
                              43200      ; refresh (12hours)
                              3600       ; retry (1hour)
                              1209600    ; expire (2week)
                              6200)      ; TTL minimum (2hour)
 
                        IN      NS      ns1.koroglu.org.
                        IN      NS      ns2.koroglu.org.
                        IN      A       2.2.2.2
 
ns1                     IN      A       1.1.1.1
ns2                     IN      A       2.2.2.2
www                     IN      A       2.2.2.2
ae                      IN      CNAME   www.koroglu.org.

Slave Name Server

named.conf configuration for slave DNS

/var/named/chroot/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Koroglu.org Slave DNS Configuration
options {
        directory "/var/named";
        pid-file "/var/run/named/named.pid";
        version "not currently available";
        recursion no;
        dump-file "/var/named/data/cache_dump.db";
        zone-statistics yes;
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-notify { 1.1.1.1; };
};
 
controls { };
 
logging {
        channel slave_log {
                file "/var/log/named/slave.log" versions 3 size 5m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default {
                slave_log;
        };
};
 
zone "." IN {
        type hint;
        file "named.ca";
};
 
zone "localhost.localdomain" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "localhost" IN {
        type master;
        file "named.localhost";
        allow-update { none; };
};
 
zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "1.0.0.127.in-addr.arpa" IN {
        type master;
        file "named.loopback";
        allow-update { none; };
};
 
zone "0.in-addr.arpa" IN {
        type master;
        file "named.empty";
        allow-update { none; };
};
 
zone "koroglu.org" {
        type slave;
        file "slaves/koroglu.org";
        masters { 1.1.1.1; };
};

No zone file created manually, because slave dns server will be taking its zone data from master.

Advices

  1. At least two DNS server should be configured
  2. Master and slave servers should not be on the same subnet, may cause single point of failure
  3. Be carefull about the file permissions in chroot environment
  4. Always check named log file in case of any problem
  5. I didn’t apply RNDC on both server cause I would like to minimize the risk

Chrooted PHP-FPM with Nginx on CentOS 6

11th July 2013 by Ali Erdinç Köroğlu

What is chroot (change root)? It’s about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. When you change root to another directory you can not access files and commands outside that directory. As you see, chroot enhances the security for the system and creates a virtual environment inside of which the application operates. If a vulnerability exist in the application or code such that an attacker can gain file system access, who would only be able to access files inside virtualized environment and the rest of the operating system and directory structure would remain inaccessable.

It’s good but chroot != security

Installation

Before installation, we should add Nginx repository to /etc/yum/repos.d. EPEL repository has Nginx too but the version is very old.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Installing..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configurations

/etc/php-fpm.d/example.conf
[example]
listen = /tmp/example.sock 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/example-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
chroot = /chroot/jailchdir = / 
php_admin_value[disable_functions] = dir,chdir,opendir,readdirphp_admin_value[error_log] = /var/log/php-fpm/example-error.log
php_admin_flag[log_errors] = on
php_admin_value[error_reporting] = E_ALL & ~E_NOTICE
 
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
/etc/nginx/conf.d/example.conf
server {
        server_name ae.koroglu.org;
        root /srv-www/example;        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/example.access.log main;
        error_log /var/log/nginx/example.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ $uri.php;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/example.sock;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;        }
}

Chroot Debugging

Beginning was simple, a classic PHP configuration error..

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings.
You are required to use the date.timezone setting or the date_default_timezone_set() function.
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier.
/etc/php.ini
[Date]
date.timezone = Europe/Istanbul

PHP requires date.timezone information so I added “Europe/Istanbul” into php.ini and copied related files into chroot environment. Rest requires tracing.. 1st step is to find example pool php-fpm processes..

[root@8bitplus ~]# ps aux | grep php | grep example
nginx    11947  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11949  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11950  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11951  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11952  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example

The minimum number of child processes can not be less than 5 for any PHP-FPM pool by default so we should trace related pool process ids.

[root@8bitplus ~]# strace -p 11947 -o chroot1.txt&
[1] 12650
[root@8bitplus ~]# Process 11947 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11949 -o chroot2.txt&
[2] 12651
[root@8bitplus ~]# Process 11949 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11950 -o chroot3.txt&
[3] 12652
[root@8bitplus ~]# Process 11950 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11951 -o chroot4.txt&
[4] 12653
[root@8bitplus ~]# Process 11951 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11952 -o chroot5.txt&
[5] 12654
[root@8bitplus ~]# Process 11952 attached - interrupt to quit

I checked strace outputs it seems that /etc/resolv.conf, /etc/hosts, /etc/host.conf, /lib64/libnss_dns-2.12.so and /usr/lib64/libsoftokn3.so files needed for PHP-FPM chroot environment. Also you should know more about those files like where they come from, dependencies etc.. This will be very important while updating the system. If any upgrade operation updates those packages, your chroot may not be working due to broken shared library dependencies.

[root@8bitplus ~]# rpm -qf /lib64/libnss_dns-2.12.so /usr/lib64/libsoftokn3.so /usr/share/zoneinfo/Europe/Istanbul 
glibc-2.12-1.107.el6.x86_64nss-softokn-3.12.9-11.el6.x86_64tzdata-2013c-2.el6.noarch

Since PHP-FPM is not in the same chroot environment with Nginx, you should either make sure that both PHP-FPM and Nginx using the same chroot (Nginx should be chrooted too) or you should create soft link from Nginx document root folder to chrooted document root folder and that’s what I did..

[root@8bitplus srv-www]# pwd
/srv-www[root@8bitplus srv-www]# ls -l
total 0
lrwxrwxrwx 1 root root 34 Jul  2 18:30 example -> /chroot/jail/srv-www/example/

We could create multiple mysqld processes that listen for connections on different Unix socket files and TCP/IP ports but practical and easy way is to use TCP socket instead of dealing with mysqld_multi configuration.

/etc/my.cnf
#skip-networking
bind-address = 127.0.0.1

Also create urandom device for chroot

[root@8bitplus srv-www]# mknod -m 644 /chroot/jail/dev/urandom c 1 9

Chroot Structure

[root@8bitplus /]# tree chroot
chroot/
└── jail
    ├── dev
    │   └── urandom
    ├── etc
    │   ├── host.conf
    │   ├── hosts
    │   ├── localtime
    │   └── resolv.conf
    ├── lib64
    │   ├── libnss_dns-2.12.so
    │   └── libnss_dns.so.2 -> libnss_dns-2.12.so
    ├── srv-www
    │   └── example
    ├── usr
    │   ├── lib64
    │   │   └── libsoftokn3.so
    │   └── share
    │       └── zoneinfo
    │           ├── Europe
    │           │   └── Istanbul
    │           └── zone.tab
    └── var
        ├── lib
        │   └── php
        │       └── session
        └── log
            └── php-fpm

In this example you can use single chroot environment with multiple PHP-FPM pool. But if you’re using WordPress or any application which requires opendir function, I would strongly recommended that to create different chroot environment for each. You could test your chroot environment with PhpSpy Shell

Nginx with PHP-FPM on CentOS 6

30th June 2013 by Ali Erdinç Köroğlu

We are going to use Nginx as a HTTP server (or you can use as reverse proxy server for HTTP, SMTP, POP3, and IMAP protocols). As a general definition, HTTP or Web server is a computer delivers web page to the request of clients using the Hypertext Transfer Protocol (HTTP). Which means that you can create and serve static HTML files while your Nginx server is running. How about generating dynamic web pages or retrieving information from a database? We need a program running on web server to change the web pages, this is called Server-side scripting. It’s about embedding scripts into HTML source codes or executing the code on the server before being delivered to the client. There are many server-side scripting languages available such as ASP, Java Server Pages, Perl, Python, PHP, Ruby etc..


A server-side scripting (PHP+MySQL) from Wikipedia

Until now, everything is okay. I’m assuming that we do have the interpreter (will cover in installation) which is PHP in our entry, but how Nginx will communicate with PHP? You can say that the way Apache does :) But how Apache is doing it? Yes with mod_cgi, mod_fcgi or mod_php.. But what are they? Mod_php is an Apache module which allows Apache to interpret PHP files, mod_cgi is an Apache module which allows execution of CGI scripts and mod_fcgid is a high performance alternative to mod_cgi or mod_cgid.

FCGI or uWSGI?

Since we have no official uWSGI package and PHP plugin in CentOS-Base or EPEL repositories for now FCGI will be our buddy. Bytheway FastCGI is a protocol for interfacing interactive programs with a web server.

PHP-FPM or Spawn-fcgi ?

There are many spawn-fcgi examples in internet which you have to deal with to create init or/and bash scripts etc. but why trying to reinvent the wheel? The newer PHP (>= 5.3.3) using the included PHP-FPM and this package is in CentOS-6 Base repository. PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation. It’s not only makes configuration of FastCGI pools, but also enhances some of the FastCGI internals and increases error reporting, script termination, etc. Let’s take a look at this simple comparison table* between the most popular methods of managing FastCGI pools.

 Description PHP Spawn-fcgi + daemontools PHP-FPM
PHP daemonization: pid file, log file, setsid(), setuid(), setgid(), chroot() (-) (+) (+)
Process Management. Ability to “gracefully” stop and start PHP workers without losing any queries. php4 (-), php5 (only graceful completion) (-) (+)
Restricting IP addresses from which requests can come from php4 (-), php5 (+) (since 5.2.2) (-) (+)
Dynamic number of processes, depending on the load (adaptive process spawning) (-) (-) in SVN, PHP 5.3.3RC1+
Starting the workers with different uid/gid/chroot/environment and different php.ini options. (No need for safe mode.) (-) (-) (+)
Logging stdout and stderr (-) (-) (+)
Ability to emergency restart all the processes in the event of an accidental destruction of the shared memory opcode cache, if using an accelerator (-) (-) (+)
Forcing the completion of process if set_time_limit() fails (-) (-) (+)
 Features
Error Header (+)
Accelerated Upload Support (+)
fastcgi_finish_request() (+)
Slowlog (with backtrace) (+)

* Taken from http://php-fpm.org/about

Unix Domain Sockets or TCP Loopback Sockets ?

This is a very good question.. We’ll cover both but before you deploy anything I recommend to read those analysis..
My choice is Unix domain sockets of course, also keep in mind that you’re connecting to MySQL through Unix socket :)

Performance Analysis of Various Mechanisms for Inter-process Communication – Kwame Wright (Cooper Union), Kartik Gopalan and Hui Kang (Binghamton University)

Unix domain sockets have proven to deliver the highest throughput when compared to the other mechanisms. While its dominance is still unclear for transfers of small amounts of data, it is otherwise the best mechanism to use within a single machine.

Unix domain sockets vs. internet sockets – Robert N M Watson

In general, the argument for implementing over TCP is that it gives you location independence and immediate portability you can move the client or the daemon, update an address, and it will “just work”. The sockets layer provides a reasonable abstraction of communications services, so it’s not hard to write an application so that the connection/binding portion knows about TCP and UNIX domain sockets, and all the rest just uses the socket it’s given. So if you’re looking for performance locally, I think UNIX domain sockets probably best meet your need. Many people will code to TCP anyway because performance is often less critical, and the network portability benefit is substantial.

nginxphpfcgi
The big picture :)

Installation

Before we begin to installation we should add Nginx repository to /etc/yum/repos.d. EPEL repo has Nginx too but it’s very old version. To add EPEL repository please read read this.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Let’s start..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configuration

We’ll just add a new config file into /etc/nginx/conf.d

/etc/nginx/conf.d/aekoroglu.conf
server {
        server_name ae.koroglu.org;
        root /aekoroglu/wordpress;
        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/aek.access.log main;
        error_log /var/log/nginx/aek.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ /index.php?$args;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/php-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /aekoroglu/wordpress$fastcgi_script_name;
        }
}

PHP-FPM comes with 2 configuration file, /etc/php-fpm.conf and /etc/php-fpm.d/www.conf. php-fpm.conf file has global definitions for FPM so there is no need to edit, we’ll modify www.conf.

/etc/php-fpm.d/www.conf
; Start a new pool named 'www'.
[ae.koroglu]
listen = /tmp/php-fpm.sock
 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/www-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session

The configurations of nginx and php-fpm for unix domain sockets, if you want to use TCP socket instead, you should change configurations as shown..

/etc/nginx/conf.d/aekoroglu.conf
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass 127.0.0.1:9000;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /var/www/html/aekoroglu/wordpress$fastcgi_script_name;
        }
/etc/php-fpm.d/www.conf
; Start a new pool named 'www'.
[www]
listen = 127.0.0.1:9000 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
listen.allowed_clients = 127.0.0.1

Before you start there is one important thing to do.. By default /var/lib/php/session directory comes with root:apache privileges. So you have to change it..

chown -R nginx:nginx /var/lib/php/session

Using Nginx as Load Balancer for Tomcat

28th June 2013 by Ali Erdinç Köroğlu

You may ask why Nginx? There are many reasons which I’ll not going to write down here, but you can read the nginx testimonials here..! So lets take a look at the big picture, what we want is a server on the internet side will load the balance for the servers on LAN side. Easy, right :P We’ll have 1 load-balancer, 2 application and 1 database server. Well, let’s get started..

nginx

Let’s get rid of not necessary extra memory killer TTYs, I always use 2 :)

/etc/sysconfig/init
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# color => new RH6.0 bootup
# verbose => old-style bootup
# anything else => new style bootup without ANSI colors or positioning
BOOTUP=color
# column to start "[  OK  ]" label in
RES_COL=60
# terminal sequence to move to that column. You could change this
# to something like "tput hpa ${RES_COL}" if your terminal supports it
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
# terminal sequence to set color to a 'success' color (currently: green)
SETCOLOR_SUCCESS="echo -en \\033[0;32m"
# terminal sequence to set color to a 'failure' color (currently: red)
SETCOLOR_FAILURE="echo -en \\033[0;31m"
# terminal sequence to set color to a 'warning' color (currently: yellow)
SETCOLOR_WARNING="echo -en \\033[0;33m"
# terminal sequence to reset to the default color.
SETCOLOR_NORMAL="echo -en \\033[0;39m"
# Set to anything other than 'no' to allow hotkey interactive startup...
PROMPT=yes
# Set to 'yes' to allow probing for devices with swap signatures
AUTOSWAP=no
# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty[1-2]
# Set to '/sbin/sulogin' to prompt for password on single-user mode
# Set to '/sbin/sushell' otherwise
SINGLE=/sbin/sushell

We are going to use CentOS 6 for our system infrastructure, so let us add EPEL and nginx repositories to /etc/yum/repos.d. For EPEL repository please read read this.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

After adding the repositories

yum clean all
yum upgrade

Installation

Let’s upgrade the system for each server

yum update

Load-balancer :

yum install nginx

Application Servers :

yum install nginx tomcat6-jsp-2.1-api tomcat6-lib tomcat6 tomcat6-admin-webapps tomcat6-webapps tomcat6-servlet-2.5-api tomcat6-el-2.1-api

I’ll not cover the database side, you can choose anything you like :)

Configuration: Load-balancer

Two configuration file is important here, /etc/nginx/nginx.conf and /etc/nginx/conf.d/default.conf

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    # --- Size Limits & Buffer Overflows --- #
    client_body_buffer_size  1K;
    client_header_buffer_size 1k;
    client_max_body_size 1k;
    large_client_header_buffers 2 1k;
 
    ## Start: Timeouts ##
    client_body_timeout   10;
    client_header_timeout 10;
    keepalive_timeout     5 5;
    send_timeout          10;
 
    tcp_nodelay on;
    gzip on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
 
    include /etc/nginx/conf.d/*.conf;
    server_names_hash_bucket_size  64;
    server_tokens off;
}
 /etc/nginx/conf.d/default.conf
upstream backend {
        ip_hash;
        server 192.168.1.11:80;
        server 192.168.1.12:80;
}
 
server {
        listen 80;
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass  http://backend;
        }
}

Important note: If you forget to add “ip_hash” into upstream when you start the load-balancer, your visitors will bounce from one application server to another. It’s not good if you’re deploying a session based mechanism into your application.

Configuration: Application Servers

Nginx first..

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
    server_tokens off;
 
    include /etc/nginx/conf.d/*.conf;
}
 /etc/nginx/conf.d/aekoroglu.conf
server {
        listen 80;
        server_name test.koroglu.org;
        root /usr/share/tomcat6/webapps/aekoroglu;
        access_log /var/log/nginx/aekoroglu.access.log main;
        error_log /var/log/nginx/aekoroglu.error error;
 
    location / {
        index index.jsp;
    }
 
    location ~ \.do$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
 
    location ~ \.jsp$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
    location ^~/servlets/* {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
}

What I did here is if any do,jsp or servlets requested they will come from Tomcat and the rest (other files except do,jsp or servlets) will be coming via Nginx. Cause it’s not Tomcat’s job to serve static files. Better way is to have an extra nginx server to serve statics files (css,js,jpg,png etc..)

And Tomcat

 /etc/tomcat/server.xml
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
 
  <Service name="Catalina">
    <Connector URIEncoding="UTF-8" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>
      <Valve className="org.apache.catalina.valves.RemoteIpValve"  />
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
      </Host>
 
        <!-- Virtual Hosts -->
        <Host name="test.koroglu.org" debug="0" unpackWARs="true">
        <Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="aekoroglu_log." suffix=".txt" timestamp="true"/>
        <Context path="" docBase="/usr/share/tomcat6/webapps/aekoroglu" debug="0" reloadable="true"/>
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="aekoroglu_log." suffix=".txt" pattern="common"/>
        </Host>
 
    </Engine>
  </Service>
</Server>

Optimization

I’ll write a documentation about network and system optimization soon..

Let’s run..

Load-balancer

/etc/init.d/nginx start

Application Servers

/etc/init.d/nginx start
/etc/init.d/tomcat6 start

If you want to start those services automatically in each reboot, you can add those services into startup services with chkconfig

chkconfig --level 3 nginx on
chkconfig --level 3 tomcat6 on

Or you can use ntsysv but its not coming with CentOS minimal installation, you can install with

yum install ntsysv

So that’s it, bon appetit.. :)

PS: I recommended all of you to choose minimal CentOS installation

Iterative Bash Script to Generate Fibonacci Series

23rd June 2013 by Ali Erdinç Köroğlu

In mathematics, the Fibonacci series are the numbers in the following integer sequence as 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144..
As you already know that recursion is a function that call itself, too many levels of recursion may crash script with a segfault. So lets write fibonacci series with iterative method.

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
l=1
n=0
for (( i=0; i<=25; i++ ))
do
    echo -n "$n "
    n=$(($n+$l))
    l=$(($n-$l))
done
echo ""

Output and time

erdinc@gauss ~ $ time ./fibo 
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025

real    0m0.009s
user    0m0.005s
sys     0m0.003s