The point of no return

Centralized Supervisor Interface: Cesi

28th December 2014 by Ali Erdinç Köroğlu

I already mentioned about supervisor in here. But how about to manage all supervisors from one web interface which has authorization and advanced process management filtering? As you know Supervisord provides a basic web interface to monitor and restart your processes where supervisor installed but XML-RPC interface and API allows you to control remote supervisor and the programs it runs. So it’s possible.. One UI to rule them all..

Here is Cesi (Centralized supervisor interface): it’s a web interface to manage supervisors from single UI, developed by Gülşah Köse and Kaan Özdinçer.

Application Dependencies

  1. Python : a programming language :)
  2. Flask : a microframework for Python
  3. SQlite : a self-contained, serverless, zero-configuration, transactional SQL database engine

I’ll cover everything step by step for CentOS7 minimal installation..
Since Flask is not in CentOS7 or EPEL repositories, I’ll install it via pip. So what is pip? Pip is a tool for installing and managing Python packages from Python Package Index repository. We also need Git (for cesi) and Nginx (you can get nginx from EPEL or from official Nginx repository)

Required CentOS packages

[root@supervisord ~]# yum install python-pip sqlite git nginx

Flask installation

[root@supervisord ~]# pip install flask

Getting a clone from github

[root@supervisord ~]# cd /opt/
[root@supervisord opt]# git clone https://github.com/Gamegos/cesi

Initial database operations

[root@supervisord cesi]# sqlite3 /opt/cesi/cesi/userinfo.db < userinfo.sql

Conf file should be in /etc

[root@supervisord cesi]# mv /opt/cesi/cesi/cesi.conf /etc

Let us define some remote supervisors

/etc/cesi.conf
[node:srv4]
username = superv
password = superv1q2w3e
host = 192.168.9.4
port = 9001
 
[node:srv10]
username = superv
password = superv
host = 192.168.9.10
port = 9001
 
[environment:glassfish]
members = srv4, srv10
 
[cesi]
database = /root/cesi/cesi/userinfo.db
activity_log = /root/cesi/cesi/cesi_activity.log

Since cesi will run by nginx user /opt/cesi should be accessible for it

[root@supervisord opt]# chown -R nginx:nginx /opt/cesi

We’ll run cesi via supervisord :)

/etc/supervisor.d/cesi.ini
[program:cesi]
command=/bin/python /opt/cesi/cesi/web.py
process_name=%(program_name)s
user=nginx
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/cesi-stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/var/log/cesi-stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

Flask will initialise twice the application in debug mode and when you try to stop via supervisord it will not stop.

root     17614  0.0  0.3 227168 12632 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17723  6.5  0.4 235768 16684 ?        S    21:36   0:00 /bin/python /opt/cesi/cesi/web.py
nginx    17728  6.5  0.4 309592 16936 ?        Sl   21:36   0:00 /bin/python /opt/cesi/cesi/web.py
[root@supervisord cesi]# supervisorctl stop cesi
cesi: stopped
[root@supervisord cesi]# ps aux
.
.
root     17614  0.0  0.3 227320 12664 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17728  0.5  0.4 309592 16936 ?        Sl   21:36   0:01 /bin/python /opt/cesi/cesi/web.py
root     17738  0.0  0.0 123356  1380 pts/0    R+   21:40   0:00 ps aux
[root@supervisord cesi]# supervisorctl stop cesi
cesi: ERROR (not running)

As you see it’s not stopped, so simplest thing would be to change use_reloader in web.py

/opt/cesi/cesi/web.py
--- web.py.org	2014-12-27 21:23:37.625143414 +0200
+++ web.py	2014-12-27 21:23:48.222118215 +0200
@@ -531,7 +531,7 @@
 
 try:
     if __name__ == '__main__':
-        app.run(debug=True, use_reloader=True)
+        app.run(debug=True, use_reloader=False)
 except xmlrpclib.Fault as err:
     print "A fault occurred"
     print "Fault code: %d" % err.faultCode

Working like a charm..

[root@supervisord cesi]# supervisorctl start cesi
cesi: started
[root@supervisord cesi]# ps aux| grep cesi
nginx    17704  0.2  0.4 237048 18232 ?        S    21:30   0:00 /bin/python /opt/cesi/cesi/web.py

System side is ok and let us configure the nginx..

/etc/nginx/conf.d/default.conf
server {
        server_name localhost 192.168.9.240 212.213.214.215;
 
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:5000;
        }
 
        location /static {
                root /opt/cesi/cesi/;
        }
}

Let’s check TCP sockets

[root@supervisord cesi]# netstat -anp| grep LISTEN       
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      17761/python        
tcp        0      0 192.168.9.240:9001      0.0.0.0:*               LISTEN      17614/python        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      17842/nginx: master 
tcp        0      0 192.168.9.240:22        0.0.0.0:*               LISTEN      852/sshd

It’s time to login..


Login screen


Dashboard


All supervisors in one screen


A node


A process log

Kudos goes to Gülşah and Kaan and that’s all folks..

Chrooted PHP-FPM with Nginx on CentOS 6

11th July 2013 by Ali Erdinç Köroğlu

What is chroot (change root)? It’s about creating a virtualized environment in Linux operating system to separate it from the main operating system and directory structure. When you change root to another directory you can not access files and commands outside that directory. As you see, chroot enhances the security for the system and creates a virtual environment inside of which the application operates. If a vulnerability exist in the application or code such that an attacker can gain file system access, who would only be able to access files inside virtualized environment and the rest of the operating system and directory structure would remain inaccessable.

It’s good but chroot != security

Installation

Before installation, we should add Nginx repository to /etc/yum/repos.d. EPEL repository has Nginx too but the version is very old.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Installing..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configurations

/etc/php-fpm.d/example.conf
[example]
listen = /tmp/example.sock 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/example-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
chroot = /chroot/jailchdir = / 
php_admin_value[disable_functions] = dir,chdir,opendir,readdirphp_admin_value[error_log] = /var/log/php-fpm/example-error.log
php_admin_flag[log_errors] = on
php_admin_value[error_reporting] = E_ALL & ~E_NOTICE
 
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
/etc/nginx/conf.d/example.conf
server {
        server_name ae.koroglu.org;
        root /srv-www/example;        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/example.access.log main;
        error_log /var/log/nginx/example.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ $uri.php;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/example.sock;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;        }
}

Chroot Debugging

Beginning was simple, a classic PHP configuration error..

Warning: date() [function.date]: It is not safe to rely on the system's timezone settings.
You are required to use the date.timezone setting or the date_default_timezone_set() function.
In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier.
/etc/php.ini
[Date]
date.timezone = Europe/Istanbul

PHP requires date.timezone information so I added “Europe/Istanbul” into php.ini and copied related files into chroot environment. Rest requires tracing.. 1st step is to find example pool php-fpm processes..

[root@8bitplus ~]# ps aux | grep php | grep example
nginx    11947  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11949  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11950  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11951  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example       
nginx    11952  0.0  0.2 344656  4496 ?        S    13:07   0:00 php-fpm: pool example

The minimum number of child processes can not be less than 5 for any PHP-FPM pool by default so we should trace related pool process ids.

[root@8bitplus ~]# strace -p 11947 -o chroot1.txt&
[1] 12650
[root@8bitplus ~]# Process 11947 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11949 -o chroot2.txt&
[2] 12651
[root@8bitplus ~]# Process 11949 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11950 -o chroot3.txt&
[3] 12652
[root@8bitplus ~]# Process 11950 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11951 -o chroot4.txt&
[4] 12653
[root@8bitplus ~]# Process 11951 attached - interrupt to quit
 
[root@8bitplus ~]# strace -p 11952 -o chroot5.txt&
[5] 12654
[root@8bitplus ~]# Process 11952 attached - interrupt to quit

I checked strace outputs it seems that /etc/resolv.conf, /etc/hosts, /etc/host.conf, /lib64/libnss_dns-2.12.so and /usr/lib64/libsoftokn3.so files needed for PHP-FPM chroot environment. Also you should know more about those files like where they come from, dependencies etc.. This will be very important while updating the system. If any upgrade operation updates those packages, your chroot may not be working due to broken shared library dependencies.

[root@8bitplus ~]# rpm -qf /lib64/libnss_dns-2.12.so /usr/lib64/libsoftokn3.so /usr/share/zoneinfo/Europe/Istanbul 
glibc-2.12-1.107.el6.x86_64nss-softokn-3.12.9-11.el6.x86_64tzdata-2013c-2.el6.noarch

Since PHP-FPM is not in the same chroot environment with Nginx, you should either make sure that both PHP-FPM and Nginx using the same chroot (Nginx should be chrooted too) or you should create soft link from Nginx document root folder to chrooted document root folder and that’s what I did..

[root@8bitplus srv-www]# pwd
/srv-www[root@8bitplus srv-www]# ls -l
total 0
lrwxrwxrwx 1 root root 34 Jul  2 18:30 example -> /chroot/jail/srv-www/example/

We could create multiple mysqld processes that listen for connections on different Unix socket files and TCP/IP ports but practical and easy way is to use TCP socket instead of dealing with mysqld_multi configuration.

/etc/my.cnf
#skip-networking
bind-address = 127.0.0.1

Also create urandom device for chroot

[root@8bitplus srv-www]# mknod -m 644 /chroot/jail/dev/urandom c 1 9

Chroot Structure

[root@8bitplus /]# tree chroot
chroot/
└── jail
    ├── dev
    │   └── urandom
    ├── etc
    │   ├── host.conf
    │   ├── hosts
    │   ├── localtime
    │   └── resolv.conf
    ├── lib64
    │   ├── libnss_dns-2.12.so
    │   └── libnss_dns.so.2 -> libnss_dns-2.12.so
    ├── srv-www
    │   └── example
    ├── usr
    │   ├── lib64
    │   │   └── libsoftokn3.so
    │   └── share
    │       └── zoneinfo
    │           ├── Europe
    │           │   └── Istanbul
    │           └── zone.tab
    └── var
        ├── lib
        │   └── php
        │       └── session
        └── log
            └── php-fpm

In this example you can use single chroot environment with multiple PHP-FPM pool. But if you’re using WordPress or any application which requires opendir function, I would strongly recommended that to create different chroot environment for each. You could test your chroot environment with PhpSpy Shell

Nginx with PHP-FPM on CentOS 6

30th June 2013 by Ali Erdinç Köroğlu

We are going to use Nginx as a HTTP server (or you can use as reverse proxy server for HTTP, SMTP, POP3, and IMAP protocols). As a general definition, HTTP or Web server is a computer delivers web page to the request of clients using the Hypertext Transfer Protocol (HTTP). Which means that you can create and serve static HTML files while your Nginx server is running. How about generating dynamic web pages or retrieving information from a database? We need a program running on web server to change the web pages, this is called Server-side scripting. It’s about embedding scripts into HTML source codes or executing the code on the server before being delivered to the client. There are many server-side scripting languages available such as ASP, Java Server Pages, Perl, Python, PHP, Ruby etc..


A server-side scripting (PHP+MySQL) from Wikipedia

Until now, everything is okay. I’m assuming that we do have the interpreter (will cover in installation) which is PHP in our entry, but how Nginx will communicate with PHP? You can say that the way Apache does :) But how Apache is doing it? Yes with mod_cgi, mod_fcgi or mod_php.. But what are they? Mod_php is an Apache module which allows Apache to interpret PHP files, mod_cgi is an Apache module which allows execution of CGI scripts and mod_fcgid is a high performance alternative to mod_cgi or mod_cgid.

FCGI or uWSGI?

Since we have no official uWSGI package and PHP plugin in CentOS-Base or EPEL repositories for now FCGI will be our buddy. Bytheway FastCGI is a protocol for interfacing interactive programs with a web server.

PHP-FPM or Spawn-fcgi ?

There are many spawn-fcgi examples in internet which you have to deal with to create init or/and bash scripts etc. but why trying to reinvent the wheel? The newer PHP (>= 5.3.3) using the included PHP-FPM and this package is in CentOS-6 Base repository. PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation. It’s not only makes configuration of FastCGI pools, but also enhances some of the FastCGI internals and increases error reporting, script termination, etc. Let’s take a look at this simple comparison table* between the most popular methods of managing FastCGI pools.

 Description PHP Spawn-fcgi + daemontools PHP-FPM
PHP daemonization: pid file, log file, setsid(), setuid(), setgid(), chroot() (-) (+) (+)
Process Management. Ability to “gracefully” stop and start PHP workers without losing any queries. php4 (-), php5 (only graceful completion) (-) (+)
Restricting IP addresses from which requests can come from php4 (-), php5 (+) (since 5.2.2) (-) (+)
Dynamic number of processes, depending on the load (adaptive process spawning) (-) (-) in SVN, PHP 5.3.3RC1+
Starting the workers with different uid/gid/chroot/environment and different php.ini options. (No need for safe mode.) (-) (-) (+)
Logging stdout and stderr (-) (-) (+)
Ability to emergency restart all the processes in the event of an accidental destruction of the shared memory opcode cache, if using an accelerator (-) (-) (+)
Forcing the completion of process if set_time_limit() fails (-) (-) (+)
 Features
Error Header (+)
Accelerated Upload Support (+)
fastcgi_finish_request() (+)
Slowlog (with backtrace) (+)

* Taken from http://php-fpm.org/about

Unix Domain Sockets or TCP Loopback Sockets ?

This is a very good question.. We’ll cover both but before you deploy anything I recommend to read those analysis..
My choice is Unix domain sockets of course, also keep in mind that you’re connecting to MySQL through Unix socket :)

Performance Analysis of Various Mechanisms for Inter-process Communication – Kwame Wright (Cooper Union), Kartik Gopalan and Hui Kang (Binghamton University)

Unix domain sockets have proven to deliver the highest throughput when compared to the other mechanisms. While its dominance is still unclear for transfers of small amounts of data, it is otherwise the best mechanism to use within a single machine.

Unix domain sockets vs. internet sockets – Robert N M Watson

In general, the argument for implementing over TCP is that it gives you location independence and immediate portability you can move the client or the daemon, update an address, and it will “just work”. The sockets layer provides a reasonable abstraction of communications services, so it’s not hard to write an application so that the connection/binding portion knows about TCP and UNIX domain sockets, and all the rest just uses the socket it’s given. So if you’re looking for performance locally, I think UNIX domain sockets probably best meet your need. Many people will code to TCP anyway because performance is often less critical, and the network portability benefit is substantial.

nginxphpfcgi
The big picture :)

Installation

Before we begin to installation we should add Nginx repository to /etc/yum/repos.d. EPEL repo has Nginx too but it’s very old version. To add EPEL repository please read read this.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

Let’s start..

yum install nginx php-common php-cli php-pdo php-mysql php-fpm

Configuration

We’ll just add a new config file into /etc/nginx/conf.d

/etc/nginx/conf.d/aekoroglu.conf
server {
        server_name ae.koroglu.org;
        root /aekoroglu/wordpress;
        add_header "X-UA-Compatible" "IE=Edge,chrome=1";
        access_log /var/log/nginx/aek.access.log main;
        error_log /var/log/nginx/aek.error error;
        index index.php ;
 
        location / {
                try_files $uri $uri/ /index.php?$args;
        }
 
        location = /favicon.ico {
                log_not_found off;
                access_log off;
        }
 
        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
        }
 
        location ~* \.(ico|css|js|gif|jpg|jpeg|png)(\?[0-9]+)?$ {
                expires max;
                log_not_found off;
                access_log off;
        }
 
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/tmp/php-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /aekoroglu/wordpress$fastcgi_script_name;
        }
}

PHP-FPM comes with 2 configuration file, /etc/php-fpm.conf and /etc/php-fpm.d/www.conf. php-fpm.conf file has global definitions for FPM so there is no need to edit, we’ll modify www.conf.

/etc/php-fpm.d/www.conf
; Start a new pool named 'www'.
[ae.koroglu]
listen = /tmp/php-fpm.sock
 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
; Unix user/group of processes
; RPM: apache Choosed to be able to access some dir as httpd
user = nginx
; RPM: Keep a group allowed to write in log dir.
group = nginx
 
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
 
; The number of child processes created on startup.
pm.start_servers = 5
 
; The desired minimum number of idle server processes.
pm.min_spare_servers = 5
 
; The desired maximum number of idle server processes.
pm.max_spare_servers = 35
 
; The log file for slow requests
slowlog = /var/log/php-fpm/www-slow.log
 
; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024
 
; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
;rlimit_core = 0
 
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 128M
 
; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session

The configurations of nginx and php-fpm for unix domain sockets, if you want to use TCP socket instead, you should change configurations as shown..

/etc/nginx/conf.d/aekoroglu.conf
        location ~ \.php$ {
                try_files $uri = 404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass 127.0.0.1:9000;                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME /var/www/html/aekoroglu/wordpress$fastcgi_script_name;
        }
/etc/php-fpm.d/www.conf
; Start a new pool named 'www'.
[www]
listen = 127.0.0.1:9000 
; Set listen(2) backlog. A value of '-1' means unlimited.
; Default Value: -1
;listen.backlog = -1
 
listen.allowed_clients = 127.0.0.1

Before you start there is one important thing to do.. By default /var/lib/php/session directory comes with root:apache privileges. So you have to change it..

chown -R nginx:nginx /var/lib/php/session

Using Nginx as Load Balancer for Tomcat

28th June 2013 by Ali Erdinç Köroğlu

You may ask why Nginx? There are many reasons which I’ll not going to write down here, but you can read the nginx testimonials here..! So lets take a look at the big picture, what we want is a server on the internet side will load the balance for the servers on LAN side. Easy, right :P We’ll have 1 load-balancer, 2 application and 1 database server. Well, let’s get started..

nginx

Let’s get rid of not necessary extra memory killer TTYs, I always use 2 :)

/etc/sysconfig/init
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# color => new RH6.0 bootup
# verbose => old-style bootup
# anything else => new style bootup without ANSI colors or positioning
BOOTUP=color
# column to start "[  OK  ]" label in
RES_COL=60
# terminal sequence to move to that column. You could change this
# to something like "tput hpa ${RES_COL}" if your terminal supports it
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
# terminal sequence to set color to a 'success' color (currently: green)
SETCOLOR_SUCCESS="echo -en \\033[0;32m"
# terminal sequence to set color to a 'failure' color (currently: red)
SETCOLOR_FAILURE="echo -en \\033[0;31m"
# terminal sequence to set color to a 'warning' color (currently: yellow)
SETCOLOR_WARNING="echo -en \\033[0;33m"
# terminal sequence to reset to the default color.
SETCOLOR_NORMAL="echo -en \\033[0;39m"
# Set to anything other than 'no' to allow hotkey interactive startup...
PROMPT=yes
# Set to 'yes' to allow probing for devices with swap signatures
AUTOSWAP=no
# What ttys should gettys be started on?
ACTIVE_CONSOLES=/dev/tty[1-2]
# Set to '/sbin/sulogin' to prompt for password on single-user mode
# Set to '/sbin/sushell' otherwise
SINGLE=/sbin/sushell

We are going to use CentOS 6 for our system infrastructure, so let us add EPEL and nginx repositories to /etc/yum/repos.d. For EPEL repository please read read this.

 nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

After adding the repositories

yum clean all
yum upgrade

Installation

Let’s upgrade the system for each server

yum update

Load-balancer :

yum install nginx

Application Servers :

yum install nginx tomcat6-jsp-2.1-api tomcat6-lib tomcat6 tomcat6-admin-webapps tomcat6-webapps tomcat6-servlet-2.5-api tomcat6-el-2.1-api

I’ll not cover the database side, you can choose anything you like :)

Configuration: Load-balancer

Two configuration file is important here, /etc/nginx/nginx.conf and /etc/nginx/conf.d/default.conf

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    # --- Size Limits & Buffer Overflows --- #
    client_body_buffer_size  1K;
    client_header_buffer_size 1k;
    client_max_body_size 1k;
    large_client_header_buffers 2 1k;
 
    ## Start: Timeouts ##
    client_body_timeout   10;
    client_header_timeout 10;
    keepalive_timeout     5 5;
    send_timeout          10;
 
    tcp_nodelay on;
    gzip on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
    gzip_buffers 16 8k;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
 
    include /etc/nginx/conf.d/*.conf;
    server_names_hash_bucket_size  64;
    server_tokens off;
}
 /etc/nginx/conf.d/default.conf
upstream backend {
        ip_hash;
        server 192.168.1.11:80;
        server 192.168.1.12:80;
}
 
server {
        listen 80;
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass  http://backend;
        }
}

Important note: If you forget to add “ip_hash” into upstream when you start the load-balancer, your visitors will bounce from one application server to another. It’s not good if you’re deploying a session based mechanism into your application.

Configuration: Application Servers

Nginx first..

 /etc/nginx/nginx.conf
user  nginx;
worker_processes  4;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
    server_tokens off;
 
    include /etc/nginx/conf.d/*.conf;
}
 /etc/nginx/conf.d/aekoroglu.conf
server {
        listen 80;
        server_name test.koroglu.org;
        root /usr/share/tomcat6/webapps/aekoroglu;
        access_log /var/log/nginx/aekoroglu.access.log main;
        error_log /var/log/nginx/aekoroglu.error error;
 
    location / {
        index index.jsp;
    }
 
    location ~ \.do$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
 
    location ~ \.jsp$ {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
    location ^~/servlets/* {
        proxy_pass http://localhost:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
    }
}

What I did here is if any do,jsp or servlets requested they will come from Tomcat and the rest (other files except do,jsp or servlets) will be coming via Nginx. Cause it’s not Tomcat’s job to serve static files. Better way is to have an extra nginx server to serve statics files (css,js,jpg,png etc..)

And Tomcat

 /etc/tomcat/server.xml
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
 
  <Service name="Catalina">
    <Connector URIEncoding="UTF-8" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>
      <Valve className="org.apache.catalina.valves.RemoteIpValve"  />
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
      </Host>
 
        <!-- Virtual Hosts -->
        <Host name="test.koroglu.org" debug="0" unpackWARs="true">
        <Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="aekoroglu_log." suffix=".txt" timestamp="true"/>
        <Context path="" docBase="/usr/share/tomcat6/webapps/aekoroglu" debug="0" reloadable="true"/>
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="aekoroglu_log." suffix=".txt" pattern="common"/>
        </Host>
 
    </Engine>
  </Service>
</Server>

Optimization

I’ll write a documentation about network and system optimization soon..

Let’s run..

Load-balancer

/etc/init.d/nginx start

Application Servers

/etc/init.d/nginx start
/etc/init.d/tomcat6 start

If you want to start those services automatically in each reboot, you can add those services into startup services with chkconfig

chkconfig --level 3 nginx on
chkconfig --level 3 tomcat6 on

Or you can use ntsysv but its not coming with CentOS minimal installation, you can install with

yum install ntsysv

So that’s it, bon appetit.. :)

PS: I recommended all of you to choose minimal CentOS installation