Tuesday, June 7, 2011

Using Nginx as reverse proxy

cat /proc/cpuinfo
yum install apt (cài gói apt-get )
yum install pcre-devel zlib-devel openssl-devel
rpm -qa|grep pcre


It’s common knowledge that when you’re serving a web application you shouldn’t use a standard Apache install to serve static assets, as it comes with too much overhead. I won’t go into the details of why here, as it’s been covered by many other people better qualified than I.

What I can do, however, is tell you how I set up Nginx, which is a super light–weight web server, on my VPS here on Slicehost (who are awesome, by the way).

Quick theory

Just quickly, the theory is that Nginx listens on port 80, and subsequently sends requests for certain URL patterns through to the mod_wsgi server (in my case, Apache) listening on a different port. This server currently serves the meat of my Django site. Static assets (JS, CSS, images) are served directly from Nginx without ever touching Apache.

Assumptions

We assume several things for this article:

  • You’re comfortable with a command line;
  • You’re using Ubuntu or Debian (I use apt-get quite a lot);
  • You have sudo access to a server; and
  • You’re already serving Django or similar on Apache and just want to replace the static/front-end.

First steps

Firstly you’ll need the basic tools to install Nginx:

sudo apt-get install libpcre3 libpcre3-dev libpcrecpp0 \
libssl-dev zlib1g-dev make

What we’re installing here is the minimum amount of tools needed to run GZip and URL re–writing with Nginx.

Get Nginx

At the time of writing, the latest stable version of Nginx was 0.6.32, so let’s get that. Note that we need full source code as the version that ships with Ubuntu and Debian is 0.5.3 or similar, which doesn’t have URL rewriting or GZip compression (both of which I really want).

mkdir ~/src
cd !$
wget http://sysoev.ru/nginx/nginx-0.6.32.tar.gz
tar -zxvf nginx-0.6.32.tar.gz
cd nginx-0.6.32

So we downloaded the source, de–compressed it, and went into the directory that was created.

Compile Nginx

We have a few different options to run here, most of which are personal taste. Feel free to modify as required:

./configure --pid-path=/var/run/nginx.pid \
--conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin \
--with-http_ssl_module --user=www-data --group=www-data \
--http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log

The only thing I would say should be kept there is the PID file path and the user/group configuration. The user/group matches the accounts that Apache uses, so it keeps everything under the same user structure. If you want to use a different user account, be sure to create this user before running ./configure.

The above command will spit out a set of paths for your convenience: these should look similar to the following:

nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/sbin"
nginx configuration prefix: "/etc/nginx"
nginx configuration file: "/etc/nginx/nginx.conf"
nginx pid file: "/var/run/nginx.pid"
nginx error log file: "/usr/local/nginx/logs/error.log"
nginx http access log file: "/usr/local/nginx/logs/access.log"
nginx http client request body temporary files: "/usr/local/nginx/client_body_temp"
nginx http proxy temporary files: "/usr/local/nginx/proxy_temp"
nginx http fastcgi temporary files: "/usr/local/nginx/fastcgi_temp"

You may want to copy them somewhere for posterity.

Then we do the usual make/make install dance.

make
sudo make install

Nginx will now have started, but won’t be running because Apache is using port 80, and Nginx is very helpful and fails silently.

Swap Apache and Nginx

First we need to stop Apache:

sudo apache2ctl stop

Then we start Nginx:

sudo /usr/local/sbin/nginx

Note that the path to nginx will be different depending on what value (if any) you used in the ./configure stage.

If you now navigate to your IP address, you should see a “Welcome to Nginx!” message. Great!

Make Apache listen on a different port

I chose port 8080, since that seemed sensible and symmetrical.

sudo vi /etc/apache2/ports.conf

And change the value to something you can remember.

sudo apache2ctl start

And navigate to your old site but with :8080 appended to the IP address. You should see your old site there. (Note: I’ve added extra information about Apache at the end of this article).

Configure Nginx

Nginx comes with some initial configuration, but here’s what I use:

# smart default nginx (Ubuntu 7.10)

user www-data www-data;
worker_processes 2;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
use epoll;
}

http {
# allow long server names
server_names_hash_bucket_size 64;

include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log;

# spool uploads to disk instead of clobbering downstream servers
client_body_temp_path /var/spool/nginx-client-body 1 2;
client_max_body_size 32m;
client_body_buffer_size 128k;

server_tokens off;

sendfile on;
tcp_nopush on;
tcp_nodelay off;

keepalive_timeout 5;

## Compression
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_types text/plain text/html text/css application/x-javascript \
text/xml application/xml application/xml+rss text/javascript;
# Some version of IE 6 don't handle compression well on some mime-types,
# so just disable for them
gzip_disable "MSIE [1-6].(?!.*SV1)";
# Set a vary header so downstream proxies don't send cached gzipped
# content to IE6
gzip_vary on;

# proxy settings
proxy_redirect off;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

include /etc/nginx/sites-enabled/*;

}

Note that this is the primary configuration, which if you’d followed the above installation verbatim would be at /etc/nginx/nginx.conf.

To test that this configuration works, we add a simple localhost configuration file:

sudo mkdir /etc/nginx/sites-enabled
sudo vi /etc/nginx/sites-enabled/localhost.conf

And put the following configuration into it:

server {
listen 80;
server_name localhost;

location / {
root html;
index index.html index.htm;
}
}

Proxy requests to Apache

Now we need to send requests to Apache. This is actually very simple:

sudo vi /etc/nginx/sites-enabled/testproject.conf

We’re pretending that your domain is at testproject.com for the purposes of this exercise.

Enter the following into your domain config:

# primary server - proxypass to Django
server {
listen 80;
server_name dev.testproject.com;

access_log off;
error_log off;

# proxy to Apache 2 and mod_python
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

client_max_body_size 10m;
client_body_buffer_size 128k;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}



}

Again, the IP address and locations of configuration files depend on whether you changed anything during the process so far.

That’s it!

When you next start Nginx, it should send all requests through to Apache on port 8080, and your memory overhead should start coming down.

What next?

In the next instalment we’re going to set up Nginx as a static content server, in order to bypass Apache completely for anything non–dynamic.

Enjoy!

Additional reading

This article is based on the hard work of those awesome people over at Slicehost, and my experience on their servers.

Update:

Gareth Rushgrove mentioned to me at work that if you’re not exposing Apache to the world on port 80, you probably shouldn’t let it listen to any interface except loopback (otherwise people can see your dynamic site on http://yourdomain.com:8080). This isn’t an issue for me because I firewall almost every port except 80, but in case you’re interested here’s how to configure Apache:

sudo vim /etc/apache2/ports.conf

And add 127.0.0.1: before the port number you’re using for your Apache, for example:

Listen 127.0.0.1:8080

Now restart Apache and you should be secure that only Nginx is receiving HTTP requests from the outside world (or “The Internets”, as we in the industry call it).

To check what interfaces are listening, period, use this command: netstat -pant.

http://tumblr.intranation.com/post/766288369/using-nginx-reverse-proxy
http://gaquay.wordpress.com/2009/04/09/cai-dat-nginx-tu-file-ma-nguon/

http://hi.baidu.com/start_and_end/blog/item/336203df7d956318632798b0.html

#wget

http://sysoev.ru/nginx/nginx-0.6.26.tar.gz
#tar zxvf nginx-0.6.26.tar.gz
#cd nginx-0.6.26
#[root@test nginx-0.6.26]# ./configure
./configure: error: the HTTP rewrite module requires the PCRE library.
You can either disable the module by using --without-http_rewrite_module
option, or install the PCRE library into the system, or build the PCRE library
statically from the source with nginx by using --with-pcre= option.
[root@test nginx-0.6.26]# rpm -qa|grep pcre
pcre-6.6-1.1
[root@test suantop]# rpm -ivh pcre-devel-6.6-1.1.i386.rpm
warning: pcre-devel-6.6-1.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing... ########################################### [100%]
1:pcre-devel ########################################### [100%]

[root@test suantop]# rpm -qa|grep pcre
pcre-6.6-1.1
pcre-devel-6.6-1.1

等再次执行./configure
[root@test nginx-0.6.26]# ./configure --with-http_stub_status_module
[root@test nginx-0.6.26]# make
[root@test nginx-0.6.26]# make install
修改配置文件 /usr/local/nginx/conf/nginx.conf
给出一些主要更改的地方
user nobody nobody;
worker_processes 2;
error_log logs/error.log notice;
pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
include /usr/local/nginx/conf/proxy.conf;
#上面这行是增加的
server {
listen 81;
server_name localhost;
charset gb2312;
# serve static files
location ~ ^/(images|pages|javascript|js|css|flash|media|static)/ {
root /usr/local/www;
expires 30d;
location / {
root /usr/local/www;
index index.jsp index.htm;
proxy_pass http://192.168.1.250:8080;
}

location /NginxStatus {
stub_status on;
access_log on;
auth_basic "NginxStatus";
}


说明配置文件中有提到proxy.conf 这个文件可以VI一个如下文件
[root@test conf]# cat proxy.conf
#!nginx (-)
# proxy.conf
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;
测试配置文件
[root@test conf]# /usr/local/nginx/sbin/nginx -t
2008/05/04 15:40:02 [info] 2363#0: the configuration file /usr/local/nginx/conf/nginx.conf syntax is

ok
2008/05/04 15:40:02 [info] 2363#0: the configuration file /usr/local/nginx/conf/nginx.conf was tested

successfully
启动
[root@test conf]# /usr/local/nginx/sbin/nginx
[root@test conf]# ps fax
9400 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx
9401 ? S 0:00 \_ nginx: worker process
9402 ? S 0:00 \_ nginx: worker process

http://192.168.1.250:81/NginxStatus
Active connections: 1
server accepts handled requests
2 2 11
Reading: 0 Writing: 1 Waiting: 0


http://192.168.1.250:81就可以访问我的页面了用着个nignx+tomcat来替代apache+jk+tomcat
同样的可以让squid来监听80端口 这样就透明代理了
查看服务器信息
[root@test conf]# curl -I http://localhost
HTTP/1.0 200 OK
Server: nginx/0.6.26
Date: Sun, 04 May 2008 07:18:27 GMT
Content-Type: text/html;charset=GBK
Set-Cookie: JSESSIONID=0CE2D90115EFDE9830215A55414BF11F; Path=/
X-Cache: MISS from test.suantop.com
Via: 1.0 test.abc.com (squid/3.0.STABLE2)
Connection: close


http://kovyrin.net/2006/04/17/typical-nginx-configurations/

Typical Configurations Overview For Nginx HTTP(S) Reverse Proxy/Web Server

In one of my previous posts I have described very powerful Unix admin tool – Nginx. As I said, main problem of this server is lack of English documentation. That is why I decided to write this post with list of typical nginx configurations and example configuration snippets for these configurations.

All sample configuration files are tested on up to date version of nginx, which has been compiled and installed with following commands:

1
2
3
# ./configure --prefix=/usr/local/nginx --with-http_ssl_module
# make
# make install

So, you can simply download my sample, rename it to nginx.conf and adjust listening/proxying settings, place conf file to /usr/local/nginx/conf/ and start your server.

Using nginx as simple web server for static files

Nginx can be easily set up as efficient web server for static files distribution. I am using this configuration in my projects on images.someproject.com sub-domains for images distribution.

Sample configuration file can be downloaded here.

Using nginx as web server with PHP support

If you need to use nginx with PHP, you can setup PHP as FastCGI and let nginx to forward all PHP queries to some FastCGI port (tcp/socket). To use this configuration you need to start PHP as FastCGI using some third party software like spawn-fcgi from lighttpd. (Notice: I am going to describe this process in one of the future posts.)

To enable PHP support, you need to add special location section to your config file:

1
2
3
4
5
6
7
8
9
10
11
12
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:12345;
fastcgi_index index.php;

fastcgi_param SCRIPT_FILENAME /usr/local/nginx/html$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
}

Sample configuration file can be downloaded here.

Using nginx as web server with SSI support

Server-Side Includes (aka SSI) is another interesting feature of nginx. As for now, following ssi instructions are supported: config, echo, if, include, set.

SSI support can be anabled by single line configuration command in your config file:

1
2
3
4
location / {
ssi on;
...
}

Sample configuration file can be downloaded here.

Using nginx as https-enabled web server

You need https-access to your Nginx-powered site? No problems! Nginx supports https and can be used to implement secured web-server with SSLv2, SSLv3 or TLSv1.

To enable https-support you should have certificate and key files. How to obtain them, you can read in SSL FAQ. When you will obtain them, you can enable ssl-module:

1
2
3
4
5
6
7
8
server {
listen 443;
ssl on;
ssl_certificate /usr/local/nginx/conf/cert.pem;
ssl_certificate_key /usr/local/nginx/conf/cert.key;
keepalive_timeout 70;
...
}

Sample configuration file can be downloaded here.

Using nginx as reverse-proxy server before some another web-server

If you have some large web-site and you have noticed, that your Apache can not handle more load, you can put nginx before your primary web-server to use it as light reverse-proxy and as web-server to handle requests to static files.

Thanks to nginx flexibility, you can pass any types of requests to backend server by using location sections (all files, only dynamic content requests or some specific locations in your web-server tree):

1
2
3
4
location / {
proxy_pass http://localhost:8000/;
proxy_set_header X-Real-IP $remote_addr;
}

Sample configuration file can be downloaded here.

Using nginx for virtual hosting platforms

One of the interesting use cases for Nginx is virtual hosting platform because it meets all requirements for good hosting server: it is efficient, it supports all popular virtual hosting methods and it has very good internal structure, so it can be easily extended in for any specific areas.

As for now, it is being used by many hosting companies as reverse proxy and I am using it on my free hosting service with millions unique visitors per day.

If you vant to try virtual hosting feature, you can create additional server sections in your config file (first section will be default):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
http {
server {
listen 192.168.10.1;
listen 192.168.10.1:8000;

server_name one.example.com www.one.example.com;
...
}

server {
listen 192.168.10.1;
listen 192.168.10.2:8000;
listen 9000;

server_name two.example.com www.two.example.com
three.example.com www.three.example.com;
...
}

server {
listen 9000;

server_name four.example.com www.four.example.com;
...
}
}

Sample configuration file for small virtual hosting can be downloaded here.

As you can see from my small overview, nginx is very flexible software and you can do many interesting things with it. If you have any comments, questions or suggestions, feel free to drop them here in comments for this article and I will try to answer for all of them.

Using Nginx As Reverse-Proxy Server On High-Loaded Sites

http://kovyrin.net/2006/05/18/nginx-as-reverse-proxy/

Two weeks ago we have started new version of one of our primary web projects and have started very massive advertisement campaign to promote this web site. As the result of that advertisements, our outgoing traffic has been increased to 200-250Mbit/s from only one server! In this article I will describe, how to build stable and efficient web site with two-layer architecture (with frontend + backend web servers) or how to modify your current server configuration to get additional resources to handle more requests.

First of all, let me describe general structure of web-server and how it handles clients requests:

  1. Client initiates request to your server.
  2. His browser connects to your server.
  3. Your server (as for Apache) creates new thread/process to handle request.
  4. If client requested dynamic content, web server spawns CGI process or executes dynamic content handling module (i.e. mod_php) and waits while request will be processed. When it receives result web-page, it sends it to client.
  5. If client asked for some static file, web server sends this file to client
  6. Client’s browser receives answer, closes connection to web server and displays content.

As you can see, when there are many requests coming to your server, your server needs to create many parallel threads/processes and keep them running while client will close connection. If client has slow connection, web server process will wait too long and resource consumption will increase very fast.

What we can do in such situation? Simple solution is to buy more memory and more CPUs for your server and wait while web server load will crash your server. But there is more efficient solution! You can simply put some small piece of software (nginx, for example) behind your big web server and let it handle all requests to static content and to pass all dynamic requests to primary web-server. With this solution your big server will spawn additional threads/processes only for static pages and it will return answers to small frontend very fast and then can free resources to use them to handle another queries. Small frontend can wait very long time while client will receive his content and will close connection – backend server will not consume resources for such long time!

Here you can see simple diagram of proposed web server configuration:


General Data Flow Diagram

As additional benefit from such configuration you can get very useful feature of managed downloads that will be described below.

If your server contains some static resources, which can be downloaded not by all users (content provider can provide mp3 files only to users with positive balance or some site can provide downloads only to logged-in users), in generic configuration you need to create some script to handle this downloads and to create some ugly links like http://some.service.com/down.php?file=xxx.mp3 and additionally your users will not be able to resume downloads (except such cases when your script so complex, that it handles Ranges HTTP-header)…

In configuration with nginx frontend, you can create simple URL-rewriting rule, that will pass all requests to pretty URLs like http://your.cool-service.com/files/cool.mp3 to some simple script /down.php automatically and, if this script has returned X-Accel-Redirect header, will send requested file to user automatically with Ranges header support and when user will download his cool content, your backend server can handle other requests. Your users even will not know that your script controls their downloads. Simple diagram for described algorithm is following:


Functional Algorithm

Let me bring your attention to interesting fact: If you only accelerate your site with described technique and do not want to create download control system, you do not need to modify any of your scripts on your backend server! They will work as in original configuration!

So, the last thing you need to boost your web server with nginx reverse proxying technique is following configuration file snipet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
server {
listen 80;
server_name some-server.com www.server-name.com;

access_log logs/host.access.log main;

# Main location
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 10m;
client_body_buffer_size 128k;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}

# Static files location
location ~* ^.+.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$ {
root /spool/www/members_ng;
}

}

Full version of sample config file you can get here.

Notice: If your backend scripts are using user IP addresses for some purposes, you will need to install mod_rpaf module to use X-Real-IP header provided by nginx instead of real user’s IP address.

That is all! Now you can simply install nginx on your server, configure it and your server will be able to handle more traffic with the less resources that it uses now! Everything will be done transparently for your currently written scripts and if you want, you will be able to provide download handling with simple trick, that I will describe in my next post ;-)

If you have some questions, do not hesitate to ask them here in comments – I will try to answer all of them. If you liked this article, you can support author by taking a look at advertisements on this page or simply vote for it on digg.com.

http://linuxtours.blogspot.com/2011/03/how-to-install-and-configure-nginx-on.html

How To Install And Configure nginx On CentOS 5.5 For Load Balancing and Caching.

I know sometimes you get a bit confused while installing or configuring nginx in order to load balance and cache.
This guide to help you install and configure it easily.
1. Enable EPEL repo:
[root@linuxguru ~]# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

2. Install nginx
[root@linuxguru ~]# yum install nginx
3. Configuring files:

  • Default config file: /etc/nginx/nginx.conf
  • Default SSL config file: /etc/nginx/conf.d/ssl.conf
  • Default virtual hosting config file: /etc/nginx/conf.d/virtual.conf
  • Default documentroot: /usr/share/nginx/html
Now, we will configure /etc/nginx/nginx.conf so that we can resever httpd port.

#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
worker_rlimit_nofile 102840;
events {
worker_connections 10024;
}

http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
access_log /dev/null ;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 60;
#gzip on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_buffering on;
proxy_cache_key "$host$request_uri";
proxy_cache_path /app/nginx/ levels=2:2 keys_zone=one:128m inactive=30d max_size=100G;
proxy_cache_valid 200 7d;
proxy_cache_valid any 1m;
proxy_temp_path /app/nginx/tmp/;
upstream webserver {
server 10.100.1.11:88 ;
server 10.100.1.12:88 ;
}
upstream cacheserver {
server 10.100.1.11:89 ;
server 10.100.1.12:89 ;
}


server {
listen 80;
server_name serverexample1.com;
access_log /dev/null ;
location / {
proxy_pass http://10.100.1.11:86;
}
}
server {
listen 80;
server_name serverexample2.com serverexample2.com;
access_log /dev/null ;
location / {
proxy_pass http://10.100.1.11:88;
}
}

server {
listen 80 ;
#server_name _;
access_log /dev/null ;
location /
{
proxy_pass http://webserver;
proxy_set_header Host $host;
}
}


}

-Remember to configure httpd.conf and change the Listen Port to 88.
- Create proxy_cache_path.



https://calomel.org/nginx.html
Nginx is a fast and efficient web server. It can be configured to serve out files or be a reverse proxy depending on your application. What makes this web server different from Apache, Lighttpd or thttpd is the overall efficiency of the daemon, the number of configuration options and how easy it is to setup.
Nginx ("engine x") is a high-performance HTTP server and reverse proxy server. Nginx was written by Igor Sysoev for rambler.ru, Russia's second-most visited website, where it has been running in production for over two and a half years. Igor has released the source code under a BSD-like license. Although still in beta, Nginx is known for its stability, rich feature set, simple configuration, and low resource consumption. Nginx


The methodology behind our configuration - Trust No One

In the following example we are going to setup a simple web server to serve our static web pages to explain the basics. The daemon will load a few mime include files, compress outgoing data in real time and set the expires header to reduce bandwidth of client cached traffic. Full logging is on, in the default Apache format with the addition of compressed file size and the amount of time the server took to fulfill the request. Finally, we are going to set up restriction filters by ip to limit access to the "/secure" directory structure where you might put more sensitive non-public data.

The security mindset of the configuration is very paranoid. There are a significant amount of bots, scanners and broken clients that will abuse your site if given the opportunity. These clients will waste your bandwidth and system resources. As a response, we will not trust any client to access our server without first making sure that all of the request parameters are met. This means that the remote client must be asking for our site by the proper host name and must request any support files, like pictures and css, with the referrer headers properly set. Any deviation from these rules will lead to Nginx dropping the client's connection with a return code 444. Even though Nginx does not have a module like mod_security we can still make our own access rules. Note that even though these rules are strict, normal web traffic and bots like Google can access the site without issue.

Our goal is to setup a fast serving and CPU/disk efficient web server, but most importantly a _very secure_ web server. This configuration will work for the latest version of Nginx as well as the development versions. For the purpose of this example we built the latest development version of Nginx from source.

Below you will find a few different example nginx.conf configuration files in scrollable windows. The formats are available to make it easier for you to review the code. They are all fully working configuration files with the exception of setting up a few variables for your environment like listen port or ip.

You are welcome to copy and paste the following working examples. Before using the configuration file take a look it and then scroll down this page to the section titled, "Explaining the directives in nginx.conf".



Option 1: Nginx webserver to serve static files

This is a basic webserver running on port 80 (http) serving out web pages. Though we have added quite a few security checks, this is as basic a server as you can get. On an AMD64 2GHz machine this config will easily serve out thousands of pages a minute.

#######################################################
### Calomel.org /etc/nginx.conf BEGIN
#######################################################
#
pid /var/run/nginx.pid;
user nginx nginx;
worker_processes 4;

events {
worker_connections 1024;
}

http {
## MIME types
types {
application/xml xml;
image/gif gif;
image/jpeg jpg;
image/png png;
image/bmp bmp;
image/x-icon ico;
text/css css;
text/html html;
text/plain bob;
text/plain txt;
}
default_type application/octet-stream;

## Size Limits
client_body_buffer_size 8k;
client_header_buffer_size 1k;
client_max_body_size 1k;
large_client_header_buffers 1 1k;

## Timeouts
client_body_timeout 60;
client_header_timeout 60;
keepalive_timeout 60 60;
send_timeout 60;

## General Options
ignore_invalid_headers on;
limit_zone gulag $binary_remote_addr 1m;
recursive_error_pages on;
sendfile on;
server_name_in_redirect off;
server_tokens off;

## TCP options
tcp_nodelay on;
tcp_nopush on;

## Compression
gzip on;
gzip_static on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.0;
gzip_min_length 0;
gzip_types text/plain text/html text/css image/x-icon image/bmp;
gzip_vary on;

## Log Format
log_format main '$remote_addr $host $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"';

## Deny access to any host other than (www.)mydomain.com
server {
server_name _; #default
return 444;
}

## Server (www.)mydomain.com
server {
add_header Cache-Control public;
access_log /var/log/nginx/access.log main buffer=32k;
error_log /var/log/nginx/error.log info;
expires 31d;
limit_conn gulag 5;
listen 127.0.0.1:8080 rcvbuf=64k backlog=128;
root /htdocs;
server_name mydomain.com www.mydomain;

## Only allow GET and HEAD request methods
if ($request_method !~ ^(GET|HEAD)$ ) {
return 444;
}

## Deny illegal Host headers
if ($host !~* ^(mydomain.com|www.mydomain.com)$ ) {
return 444;
}

## Deny certain User-Agents (case insensitive)
## The ~* makes it case insensitive as opposed to just a ~
if ($http_user_agent ~* (Baiduspider|Jullo) ) {
return 444;
}

## Deny certain Referers (case insensitive)
## The ~* makes it case insensitive as opposed to just a ~
if ($http_referer ~* (babes|click|diamond|forsale|girl|jewelry|love|nudit|organic|poker|porn|poweroversoftware|sex|teen|video|webcam|zippo) ) {
return 444;
}

## Redirect from www to non-www
if ($host = 'www.mydomain.com' ) {
rewrite ^/(.*)$ http://mydomain.com/$1 permanent;
}

## Stop Image and Document Hijacking
location ~* (\.jpg|\.png|\.css)$ {
if ($http_referer !~ ^(http://mydomain.com) ) {
return 444;
}
}

## Restricted Access directory
location ^~ /secure/ {
allow 127.0.0.1/32;
allow 10.10.10.0/24;
deny all;
auth_basic "RESTRICTED ACCESS";
auth_basic_user_file /var/www/htdocs/secure/access_list;
}

## Only allow these full URI paths relative to document root. If you only want
## to reference the filename use $request_filename instead of $request_uri
location / {
if ($request_uri ~* (^\/|\.html|\.jpg|\.org|\.png|\.css|favicon\.ico|robots\.txt)$ ) {
break;
}
return 444;
}

## Serve an empty 1x1 gif _OR_ an error 204 (No Content) for favicon.ico
location = /favicon.ico {
#empty_gif;
return 204;
}

## System Maintenance (Service Unavailable)
if (-f $document_root/system_maintenance.html ) {
error_page 503 /system_maintenance.html;
return 503;
}

## All other errors get the generic error page
error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 495 496 497
500 501 502 503 504 505 506 507 /error_page.html;
location /error_page.html {
internal;
}
}
}
#
#######################################################
### Calomel.org /etc/nginx.conf END
#######################################################




Option 2: Nginx serving only SSL and redirecting http to https

This example configuration is for a webserver that serves out SSL (https) traffic only. We will redirect all clients that try to goto port 80 (http) to port 443 (https) with a permanent 301 redirect. This type of redirect works for Google bot and other search bots too. We will also stop illegal linking and document hijacking. On a AMD64 2GHz machine this config can serve out thousands of fully encrypted https sessions per minute without a problem.

If you need help with setting up a SSL certificate with a certificate authority like GoDaddy check out the section below titled, "How to setup a GoDaddy Turbo SSL Certificate for Nginx". If you want to learn more about SSL in general then check out our Guide to Webserver SSL Certificates.

#######################################################
### Calomel.org /etc/nginx.conf BEGIN
#######################################################
pid /var/run/nginx.pid;
user nginx nginx;
worker_processes 4;

events {
worker_connections 1024;
}

http {
## MIME types
types {
application/xml xml;
image/jpeg jpg;
image/png png;
image/x-icon ico;
text/css css;
text/html html;
text/plain bob;
text/plain txt;
}
default_type application/octet-stream;

## Size Limits
client_body_buffer_size 16k;
client_header_buffer_size 1k;
client_max_body_size 1k;
large_client_header_buffers 1 1k;

## Global SSL options
ssl_ciphers HIGH:!ADH:!MD5;
#ssl_engine aesni;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;

## Timeouts
client_body_timeout 60;
client_header_timeout 60;
keepalive_timeout 60 60;
send_timeout 60;

## General Options
ignore_invalid_headers on;
keepalive_requests 10;
recursive_error_pages on;
server_tokens off;
server_name_in_redirect off;
sendfile on;

## TCP options
tcp_nopush on;
tcp_nodelay on;

## Compression
gzip on;
gzip_static on;
gzip_buffers 16 8k;
gzip_http_version 1.0;
gzip_comp_level 6;
gzip_min_length 100;
gzip_types text/plain text/html text/css image/x-icon image/gif;
gzip_vary on;

## Log Format
log_format main '$remote_addr $host $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"';

## Redirect http to https
server {
add_header Cache-Control "public, must-revalidate";
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log info;
expires 90d;
listen 127.0.0.1:80;
root /var/empty;
server_name example.com www.example.com;

location / {
if ($host ~* ^(example\.com|www\.example\.com)$ ) {
rewrite ^/(.*)$ https://example.com/$1 permanent;
}
return 444;
}
}

## https .:. "default blank SSL server, SNI required"
(look below for the tutorial titled, "default blank SSL server")
#server {
# add_header Cache-Control "public, must-revalidate";
# add_header Strict-Transport-Security "max-age=7776000; includeSubdomains";
# expires 90d;
# listen 127.0.0.1:443 default;
# return 444;
# root /var/empty;
# server_name _;

# ## SSL Certs (specific to this blank certificate)
# ssl on;
# ssl_certificate ssl_keys/default_blank.crt;
# ssl_certificate_key ssl_keys/default_blank.key;
# return 403;
# }

## https .:. (www.)example.com ONLY
server {
access_log /var/log/nginx/access.log main;
add_header Cache-Control "public, must-revalidate";
error_log /var/log/nginx/error.log info;
expires 90d;
index index.html;
listen 127.0.0.1:443;
root /var/www/htdocs;
server_name example.com www.example.com;

## SSL Certs (specific to this URL)
ssl on;
ssl_certificate /ssl_keys/mydomain.com_ssl.crt;
ssl_certificate_key /ssl_keys/mydomain_ssl.key;

## Strict Transport Security (ForceHTTPS)
add_header Strict-Transport-Security "max-age=2592000; includeSubdomains";

## Only allow GET and HEAD request methods
if ($request_method !~ ^(GET|HEAD)$ ) {
return 444;
}

## Deny illegal Host headers
if ($host !~* ^(example.com|www.example.com)$ ) {
return 444;
}

## Deny certain User-Agents (case insensitive)
# if ($http_user_agent ~* (Baiduspider|webalta|Wget|WordPress|youdao) ) {
# return 444;
# }

## Deny certain Referers (case insensitive)
if ($http_referer ~* (\.us$|dating|diamond|forsale|girl|jewelry|nudit|organic|poker|porn|poweroversoftware|sex|teen|webcam|zippo|zongdo) ) {
return 444;
}

## Only allow these full URI paths relative to document root. If you only want
## to reference the filename use $request_filename instead of $request_uri
if ($request_uri !~* (^\/|\.html|\.gif|\.jpg|\.png|example\.css|robots\.txt|favicon\.ico)$ ) {
return 444;
}

## Redirect from www to non-www
if ($host = 'www.example.com' ) {
rewrite ^/(.*)$ https://example.com/$1 permanent;
}

## Stop Image and Document Hijacking
location ~* (\.jpg|\.gif|\.png|example\.css)$ {
if ($http_referer !~ ^(https://example.com) ) {
return 444;
}
}

## Restricted Access directory
location ^~ /secure/ {
allow 127.0.0.1/32;
allow 10.10.10.0/24;
deny all;
auth_basic "RESTRICTED ACCESS";
auth_basic_user_file /var/www/htdocs/secure/access_list;
}

## All other errors get the generic error page
error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 495 496 497
500 501 502 503 504 505 506 507 /error_page.html;
location /example_error_page.html {
internal;
}
}
}
#######################################################
### Calomel.org /etc/nginx.conf END
#######################################################




Option 3: Nginx reverse proxy for back end web servers

This config is for a reverse proxy server in front of three back end web servers. One is for web content, one is a forum and the last is a file server.

As requests come in the Nginx Proxy server will look at the URL path and direct requests to the proper back end server. This config will also cache requests to the one web content server, but not the forum or data download servers. We also configured Nginx to compress http calls back to the client in real time, thus saving bandwidth.

#######################################################
### Calomel.org /etc/nginx.conf BEGIN
#######################################################
pid /var/run/nginx.pid;
user nginx nginx;
worker_processes 10;

events {
worker_connections 1024;
}

http {
## MIME types
#include /etc/nginx_mime.types;
default_type application/octet-stream;

## Size Limits
client_body_buffer_size 128K;
client_header_buffer_size 128K;
client_max_body_size 1M;
large_client_header_buffers 1 1k;

## Timeouts
client_body_timeout 60;
client_header_timeout 60;
expires 24h;
keepalive_timeout 60 60;
send_timeout 60;

## General Options
ignore_invalid_headers on;
keepalive_requests 100;
limit_zone gulag $binary_remote_addr 5m;
recursive_error_pages on;
sendfile on;
server_name_in_redirect off;
server_tokens off;

## TCP options
tcp_nodelay on;
tcp_nopush on;

## Compression
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 6;
gzip_http_version 1.0;
gzip_min_length 0;
gzip_types text/plain text/css image/x-icon application/x-perl application/x-httpd-cgi;
gzip_vary on;

## Log Format
log_format main '$remote_addr $host $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'"$gzip_ratio"';

## Proxy options
proxy_buffering on;
proxy_cache_min_uses 3;
proxy_cache_path /usr/local/nginx/proxy_temp/ levels=1:2 keys_zone=cache:10m inactive=10m max_size=1000M;
proxy_cache_valid any 10m;
proxy_ignore_client_abort off;
proxy_intercept_errors on;
proxy_next_upstream error timeout invalid_header;
proxy_redirect off;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_connect_timeout 60;
proxy_send_timeout 60;
proxy_read_timeout 60;

## Backend servers (web1 is the primary and web2 will come up if web1 is down)
upstream webbackend {
server web1.domain.lan weight=10 max_fails=3 fail_timeout=30s;
server web2.domain.lan weight=1 backup;
}

server {
add_header Cache-Control public;
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
index index.html;
limit_conn gulag 50;
listen 127.0.0.1:80 default;
root /usr/local/nginx/html;
server_name _;

## Only requests to our Host are allowed
if ($host !~ ^(mydomain.com|www.mydomain.com)$ ) {
return 444;
}

## Only allow these request methods
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
return 444;
}

## Only allow these full URI paths relative to document root. If you only want
## to reference the filename use $request_filename instead of $request_uri
location / {
if ($request_uri ~* (^\/|\.html|\.jpg|\.pl|\.png|\.css|\.ico|robots\.txt)$ ) {
break;
}
return 444;
}

## PROXY - Forum
location /forum/ {
proxy_pass http://forum.domain.lan/forum/;
}

## PROXY - Data
location /files/ {
proxy_pass http://data.domain.lan/;
}

## PROXY - Web
location / {
proxy_pass http://webbackend;
proxy_cache cache;
proxy_cache_valid 200 24h;
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
proxy_ignore_headers Expires Cache-Control;
}

## All other errors get the generic error page
error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 495 496 497
500 501 502 503 504 505 506 507 /error_page.html;
location /error_page.html {
internal;
}
}
}
#
#######################################################
### Calomel.org /etc/nginx.conf END
#######################################################




For more information about OpenBSD's Pf firewall, CARP and HFSC quality of service options check out our PF Config (pf.conf), PF CARP and PF quality of service HFSC "how to's".


Building nginx from source

To get started, you need to first install nginx on your machine. The source code is available from the nginx home page and practically every distribution has pre-made packages if you prefer those. The install is very easy and it will not take you much time.

We highly recommend you build Nginx from source. This way you can modify the code if you need to and make sure you apply the latest patches when they come out.

You need to make sure that the package for PCRE is installed. Use the command "pkg_add -i pcre" to install from your chosen PKG_PATH repository. BTW, you may want to also look at the Perl script pkg_find for OpenBSD package management.

OPTIONAL: Change the Server: string of your host

The Server: string is the header which is sent back to the client to tell them what type of http server you are running and possibly what version. This string is used by places like Alexia and Netcraft to collect statistics about how many and of what type of web server are live on the Internet. To support the author and statistics for Nginx we recommend keeping this string as is. But, for security you may not want people to know what you are running and you can change this in the source code. Edit the source file src/http/ngx_http_header_filter_module.c at look at lines 48 and 49. You can change the String to anything you want.

## vi src/http/ngx_http_header_filter_module.c (lines 48 and 49)
static char ngx_http_server_string[] = "Server: MyDomain.com" CRLF;
static char ngx_http_server_full_string[] = "Server: MyDomain.com" CRLF;

OPTIONAL: annonomize you server string in the auto generated error pages

When nginx sends an error back to the client it can auto generate the error page. This error page has the error code at the top, a single horizontal line and then the string "nginx" and possibly the version number. If you want to you can take out the server string in the error page by editing the source code in the file src/http/ngx_http_special_response.c on lines 21 and 28. The following line would make the nginx generated error pages show your domain name for example.

## vi src/http/ngx_http_special_response.c (lines 21 and 28)
"
http://mydomain.org
" CRLF

## You can also change all of the built in error
## messages with just a carriage return.
static char ngx_http_error_301_page[] = CRLF;
static char ngx_http_error_302_page[] = CRLF;
static char ngx_http_error_400_page[] = CRLF;
static char ngx_http_error_401_page[] = CRLF;
static char ngx_http_error_402_page[] = CRLF;
static char ngx_http_error_403_page[] = CRLF;
static char ngx_http_error_404_page[] = CRLF;
static char ngx_http_error_405_page[] = CRLF;
static char ngx_http_error_406_page[] = CRLF;
static char ngx_http_error_408_page[] = CRLF;
static char ngx_http_error_409_page[] = CRLF;
static char ngx_http_error_410_page[] = CRLF;
static char ngx_http_error_411_page[] = CRLF;
static char ngx_http_error_412_page[] = CRLF;
static char ngx_http_error_413_page[] = CRLF;
static char ngx_http_error_414_page[] = CRLF;
static char ngx_http_error_415_page[] = CRLF;
static char ngx_http_error_416_page[] = CRLF;
static char ngx_http_error_495_page[] = CRLF;
static char ngx_http_error_496_page[] = CRLF;
static char ngx_http_error_497_page[] = CRLF;
static char ngx_http_error_500_page[] = CRLF;
static char ngx_http_error_501_page[] = CRLF;
static char ngx_http_error_502_page[] = CRLF;
static char ngx_http_error_503_page[] = CRLF;
static char ngx_http_error_504_page[] = CRLF;
static char ngx_http_error_507_page[] = CRLF;

This same file contains all of the default HTML error pages Nginx will send to the client if there is an error. Look for the functions that start with the line static char ngx_http_error_ and make any changes you find necessary. Note that the HTML text is only shown to the user and that all errors sent by Nginx will have the proper error code in the HTML headers. This means you can put anything you want into the HTML code.

OPTIONAL: change any of the default error codes

Normally you DO NOT want to change any of the standard error codes specified by RFC. But, in case you really need to you can edit the file src/http/ngx_http_request.h and look for the variables starting with NGX_HTTP_REQUEST. For example, if we wanted to change the default error code for REQUEST_URI_TOO_LARGE from 414 to 999 we could:

vi src/http/ngx_http_request.h (line 83)
OLD LINE: #define NGX_HTTP_REQUEST_URI_TOO_LARGE 414
NEW LINE: #define NGX_HTTP_REQUEST_URI_TOO_LARGE 999

Compiling the code

Building nginx for the AMD64 architecture on OpenBSD (running as user/group "nginx")

For the purpose of this example, Nginx was built with the following arguments. Make sure to check if you need to use one of the modules that we omit during the build. We use the methodology, "if you do not need it then do not build it in." Our example nginx.conf (option 1 and 2) works fine with the following:

make clean

./configure --with-cpu-opt=amd64 --with-http_ssl_module --with-http_gzip_static_module /
--without-http_autoindex_module --without-http_browser_module --without-http_fastcgi_module /
--without-http_geo_module --without-http_empty_gif_module --without-http_map_module /
--without-http_proxy_module --without-http_memcached_module --without-http_ssi_module /
--without-http_userid_module --prefix=/usr/local/nginx --sbin-path=/usr/local/sbin/nginx /
--conf-path=/etc/nginx.conf --pid-path=/var/run/nginx.pid --http-log-path=/var/log/nginx/access.log /
--error-log-path=/var/log/nginx/error.log --user=nginx --group=nginx

make && make install

Once Nginx is built and installed in place it is time to take a look at the config file.





Explaining the directives in nginx.conf

Now we need to edit the config file for your environment. Lets take a look at each of the directives that need attention.

pid /var/run/nginx.pid : This is the location of the process id file that holds the pid number of the master Nginx process. If you wanted to re-read the nginx.conf file without restarting the daemon you could cat this file and send a HUP like so, "kill -HUP `cat /var/run/nginx.pid` .

user nginx nginx : Is the user and group the child processes will run as. You may need to make this user and group if you install Nginx from source. Make sure this user is completely unprivileged or at least runs with the least privileges necessary to make the server work.

worker_processes 2 : Is the number of worker processes to spawn. A worker is similar to a child process in Apache. Nginx has the ability to use more then one worker process for several reasons: use on (SMP) multiple processors machines, to decrease latency when workers are blocked by disk I/O, or to limit the number of connections per process when select() or poll() is used. The general rule of the thumb is to set the number of nginx workers to two(2) or the number of CPUs your server has; which ever is greater. But, on most servers you will find out that two(2) workers serve pages quickly and put less load on the server. The exception to this rule is if you use ssl and/or compress all of your content. If you use ssl and compression then we suggest testing you site with double the amount of workers. Our example nginx.conf has 2 workers so we would set it to 4.

For testing, we suggest using the Apache benchmark binary (ab) to stress your server and see how many connections your machine can handle. "ab" can be found in any Apache install. To calculate how many total concurrent connections nginx can support, multiply "worker_processes" times "worker_connections". Our example is setup to handle 2*1024=2048 total concurrent connections. Clients who attempt to connect after 2048 clients are already connected will be denied access. It is better to deny clients than overload the machine possibly causing a DOS.

worker_connections 1024 : This is the amount of client connections a single child process will handle by themselves at any one time. (default: 1024) Note: Multiply worker_processes times worker_connections for the total amount of connections Nginx will handle. Our example is setup to handle 2*1024=2048 connection in total. Clients who connect after the max has been reached will be denied access.

MIME types : This section allows nginx to identify files by extension. For example, if we serve out a .txt file then the mime type would be defined as text/plain.

include mime.types is the definition file nginx loads to identify all of the mime types. These directive simply allow our server to send the the proper file type and application type to the clients. Alternatively you can take out this line and instead define your own Mime types by using the following "type" directive".

types {...} Instead of using the "include mime.types" directive you can define your own mime types. This is especially useful option if you want to use the same mime types on many different systems or do not want to rely on a secondary definition file. You also have the option of defining a mime type for a non-standard extension. In our example we define the extension "bob" as a text/plain.

default_type application/octet-stream is the default type if a file extension has not already be defined in the mime.types file. This is useful if you serve out files with no extension or of a non standard extension. Either way, clients will be able to retrieve the file un-obstructed.

Size Limits : These directive specify the buffer size limitations on the amount of data we will consider to be valid for a request. If the client sends to much data in one request, for example in a buffer overflow attack, then the request will be denied.

client_body_buffer_size 1k If the request body is more than the buffer, then the entire request body or some part is written in a temporary file.

client_header_buffer_size 1k is the limit on the size of all of the http headers the client can send to the server. For the overwhelming majority of requests a buffer size of 1K is sufficient. The only time you would need to increase this is if you have a custom header or a large cookie sent from the client.

client_max_body_size 1k is the maximum accepted body size of client request, indicated by the line "Content-Length" in the header of request. If size exceeds this value the client gets sent the error "Request Entity Too Large" (413). If you expect to receive files uploaded to your server through the POST request method you should increase this value.

large_client_header_buffers 1 1k is the limit of the URI request line which can not be larger than the buffer size multiplied by the amount of buffers. In our example we accept a buffer size of 1 kilobyte and there is only one(1) buffer. So, will not accept a URI which is larger than (1x1K=1K) 1 kilobyte of data. If the client sends a bigger request then Nginx will return an error "Request URI too large" (414). The longest header line of the request must also be less than the size of (1x1K=1K) 1 kilobyte, otherwise the client get the error "Bad request" (400). Limiting the client URI is important to keep a scanner or broken client from sending large requests and possibly cause a denial of service (DOS) or buffer overflow.

Timeouts : These values specify the amount of time in seconds that Nginx will wait for the client to complete the specified action.

client_body_timeout 60 is the read timeout for the request body from client. If after this time the client sends nothing, nginx returns error "Request time out" (408).

client_header_timeout 60 is the timeout reading the title of the request of the client. If after this time the client send nothing, nginx returns error "Request time out" (408).

keepalive_timeout 60 60 the first value is for keep-alive connections with the client. The second parameter assigns the value "Keep-Alive: timeout=time" in the header of answer.

send_timeout 60 is response timeout to the client. Timeout is established not on the entire transfer of answer, but only between two operations of reading, if after this time client will accepts nothing, then nginx is shutting down the connection.



Want more speed? Make sure to also check out the Network Speed and Performance Guide. With a little time and understanding you could easily double your firewall's throughput.


General Options :

ignore_invalid_headers on throws away non-standard headers in the client request. If you do not expect to receive any custom made headers then make sure to enable this option.

limit_zone gulag $binary_remote_addr 1m sets up a table we will call "gulag" which uses no more than 1 megabyte of ram to store session information keyed by remote ip address. This directive is used in conjunction with limit_conn gulag 5. The ngx_http_limit_zone_module only restricts the amount of connections from a single ip address which are currently being processed by the Nginx daemon. An error 503 will be returned to the client if request processing is being blocked at the socket level and new requests from same ip starts. limit_zone will _NOT_ help if your workload is CPU or disk bound. With several workers enabled an error 503 will be returned if two workers process requests from the same ip at the same time. But this is unlikely to happen with small requests.

This is _NOT_ a directive to limit the total number of open, "established" connections to the server per ip address!! You could use your iptables or PF firewall to limit the total amount of connections. The OpenBSD PF firewall (pf.conf) uses max-src-conn or max-src-states to limit the amount of established connections to your server.

You can increase the size of the "gulag" table from 1 megabyte if you need to. A zone size of 1M can handle 32000 sessions at a default size of 32 bytes/session. You can also change the name of the table we called "gulag" to any string you want. We thought this was a good name due to Nginx's country of origin combined with the purpose of this directive.

The HTTP 1.1 specification, circa 1999, recommends that browsers and servers limit parallel requests to the same hostname to two. Most browsers comply with the multi-threading recommendation of the specification, although downgrading to HTTP 1.0 boosts parallel downloads to four. So most web browsers are effectively throttled by this limit on parallel downloads if the objects in the web page they download are hosted on one hostname. We set this limit to 5 so browsers can open 4 connections with one slot left over as a buffer. Download accelerators can open many hundreds of connections to download a file so this directive will help to alleviate abuses.

recursive_error_pages on allows the use of the error_pages directive specified later in the config.

sendfile on enables the use of sendfile(). This function can greatly increase overall system performance since sendfile() can do an entire data transfer without switching context. Enable if you allow downloads of large to medium sized files. You need to be careful about using sendfile if the file being sent has any possibility of being modified ( especially truncated ) while the operation is in progress since some very odd things ( like the process crashing ) can happen on some platforms.

server_name_in_redirect off turns off the server's ability to substitute the client supplied "Host" header with the virtual server variable "server_name" when a client is redirected.

server_tokens off turns off the nginx version numbers in the auto generated error pages. We do not want to display this information for security purposes.

TCP options : These options say how we should use the TCP stack.

tcp_nodelay on TCP_NODELAY is for a specific purpose; to disable the Nagle buffering algorithm. It should only be set for applications that send frequent small bursts of information without getting an immediate response, where timely delivery of data is required (the canonical example is mouse movements).

tcp_nopush on If set, don't send out partial frames. All queued partial frames are sent when the option is cleared again. This is useful for pre-pending headers before calling sendfile(2), or for throughput optimization. As currently implemented, there is a 200 millisecond ceiling on the time for which output is corked by TCP_CORK. If this ceiling is reached, then queued data is automatically transmitted.

Compression : These values tell nginx how to compress outgoing data. Remember that all files of the specified mime.type (gzip_types) are compressed in real time. On a P3 500MHz a 100KB HTML file takes 0.05 seconds (5 hundredths of a second) to gzip at level 9 compression (highest).

gzip on turn compression on.

gzip_static on; allows one to have pre-compressed .gz files served instead of compressing files on the fly. This is the most efficient method of serving compressed data. To use this option simply have a compressed copy of the same .html file in document root. For example, if we have the index.html file in place we will also have a pre-compressed index.html.gz file.

The following script will publish a compressed gzip file from a given html file. When you are done editing the html file execute this script to make a compressed copy ready for distribution. As soon as it is in place Nginx will serve it out. Also, make sure the date on the compressed .gz is always newer or equal to the original as Nginx will always serve out the most recent copy:

#!/bin/sh
#
## Calomel.org publish_html2gz.sh
## usage: ./publish_html2gz.sh index.html
#
## Make a tmp copy of the original HTML file
cp $1 $1.tmp

## Remove the old gz if there is one
rm -rf $1.gz

## Compress the tmp HTML copy. Use the highest level 9
## compression and do not store dates or file names
## in the gzip header. BTW, if the compressed gz is
## larger then the original file a gzip will NOT be made.
gzip -9 -n $1.tmp -o $1.gz

## Clean up any tmp files
rm -rf $1.tmp

echo ""
echo "Verify files"
ls -al $1*

echo ""
echo "Compression statistics"
gzip -vl $1.gz

When Nginx sees the .gz file it will send this out to clients who accept compression instead of compressing the file in real time. Make sure you have built your nginx binary with the argument "--with-http_gzip_static_module". Execute "nginx -V" to see the compiled options.

gzip_buffers 16 8k allows 16 slots of 8k buffers used to respond to clients with a gzip'd response. This means the max size of our compressed responses can be no larger than 16*8= 128 kilobytes. By default Nginx limits compressed responses to 4*8k= 32 kilobytes. If you expect to return responses which compressed size is more than 32KB in size then increase the number of buffers (e.g. 16). The single buffer size of 8K can not be increased.

gzip_comp_level 9 compresses files to the highest compression level. Level 1 is the fastest/lowest compression and level 9 is the slowest/best compression. During testing the time difference between level 1 and 9 was around 2 hundredths of a second per file on a P3 500MHz.

Which compression ratio is right for your server? As a test we took a standard 68.3 kilobyte HTML file and compressed it on a AMD64 1GHz machine using gzip levels 1, 6, and 9. Level 1 compressed the file 61.5%, but Level 9 took twice as long to compress the file to 67.1%. Level 1 has the best compression to time ratio. Realistically, the times are so short we still suggest using level 9, or at least level 6 compression to save overall bandwidth. Today's computers are fast enough that a user is unlikely to notice slightly more CPU usage compared to longer download times.

gzip     ratio  time      compressed  uncompressed
level 1 61.5% 0m0.009s 26320 68372
level 6 67.0% 0m0.016s 22560 68372
level 9 67.1% 0m0.018s 22525 68372

gzip_http_version 1.0 allows the server to send compressed data to HTTP/1.0 clients. HTTP/1.1 clients use the proper headers so they can always ask for compressed data.

gzip_min_length 0 this means that nginx should compress all files no matter what the size. The value is the size in bytes. You can always set this value to something higher if you do not wish to compress small files.

gzip_types text/plain text/html text/css image/bmp are the only files types to be compressed. For example, JPG's are already compressed so it would be useless for us to try to compress them again. TXT and BMP files on the other hand compress very well at an average of 250% smaller. Smaller files mean less bandwidth used and less time to transmit the same amount of data. This makes your site "feel" significantly faster.

gzip_vary on enables the response header "Vary: Accept-Encoding". This way clients know that our server has the ability to send out compressed data.

log_format main : is the log format of the web logs. This format is assigned to the variable "main" and can be used later in the http section. This format is fully compatible with standard log analyzing tools like Awstats, Webalizer and custom tools like the Calomel.org Web Log Sentry. We also have added two(2) more fields at the end of each log line. "$request_time" logs how much time the server took to generate the content and "$gzip_ratio" shows what X-factor the file was compressed by. A value of 2.50 means the file was compressed 250%.

access_log and error_log : are the locations you want the logs to be placed in. In the access_log directive you can also use the buffer command. This will buffer the access log activity into ram and once the limit has been reached Nginx will then write the logs. This can save I/O stress and bandwidth on your hard drive. You will want to remove "buffer=32k" while testing else you will not see any log output until at least 32 kilobytes of data are ready to be written to the access_log file. 32K of logs input is approximately 150 lines. The info directive on the error_log will increase the verbosity of the logs to include the reasons that clients were denied access.

expires 31d : says we want our pages to be expired from the clients cache in 31 days. Time in the Expires header is obtained as the sum of the current system time added to the time assigned in this directive. In effect, we are saying that pages are to be expired 31 days after they were accessed by the client. You can also specify a time in hours using "h". In the Nginx v0.7.0 release you can use the format "expires modified +1d" to set the expires header based on the modified time of a file. The expire header tag will tell clients they should keep a copy of the object they already downloaded for the specified amount of time. This saves a significant amount of upload bandwidth for you. Instead of clients going from page to page downloading the same picture banner over and over again, they can keep a copy locally and just get the changes on your site. Imagine a client getting 5 pages from your site. Each page has a banner that is 15KB. With expires headers enabled that client will only download the banner once instead of 5 times (15KB compared to 75KB) saving your upload bandwidth and making your site "feel" quicker responding.

limit_conn gulag 5 : limits remote clients to no more than 5 concurrently "open" connections per remote ip address being processed by Nginx. See the complimentary directive limit_zone gulag $binary_remote_addr 1m above for more information about defining the "gulag" table.

listen 127.0.0.1:8080 default rcvbuf=64K backlog=128 : tells nginx to listen on localhost (127.0.0.1) port 8080. The directive rcvbuf=64K buffers incoming data (sysctl net.inet.tcp.sendspace). rcvbuf can be decreased to as little as 1K, possible decreasing the probability of overflow during a DDoS attack. The directive backlog (sysctl kern.somaxconn) are the max number of backlogged client requests Nginx will process. If you server is quite busy you will want to increase this value. We listen on 127.0.0.1:8080 in order to use the redirection rules in iptables or in OpenBSD's pf packet filter firewall. The argument "default" says that this server {...} function should handle any client request sent to this port no matter the hostname (not used in the example).

root /var/www/htdocs : is the location of document root on your server. This is where nginx will look for all files to be served out to clients.

server_name mydomain.com www.mydomain : means this server {...} block will only answer requests that have "mydomain.com" or "www.mydomain" host headers. By default the hostname of the machine is used. We are expecting the client to ask for the correct hostname with the Host header, if not, the default server block with "server_name _;" returns an error 444 to the client. BTW, the server_name_in_redirect and server_name directives work in conjunction with each other.

SSL Options (only enable if you use a SSL certificate) If you are interested in setting up a SSL certificate for encrypted traffic on your site then we highly suggest reading our Guide to Webserver SSL Certificates. Once you understand the details of SSL certs then you must build Nginx from source and enable the argument "./configure --with-http_ssl_module".

ssl on; Enables the use of the ngx_http_ssl_module once it has been built into the Nginx binary.

ssl_certificate /ssl_keys/mydomain.com_ssl.crt; This file is the combined certificate which contains both of the "crt" files signed and sent to you by your certificate authority. See "How to setup a GoDaddy Turbo SSL Certificate on Nginx" below for details.

ssl_certificate_key /ssl_keys/mydomain_ssl.key; Specifies the location of the file with the secret key in PEM format for this server. This file is the public certificate secret key you made using the OpenSSL binary.

ssl_ciphers HIGH:!ADH:!MD5; says that our server will only accept SSL handshakes (pre-master) using AES 128/256 bit or 3DES 168 bit encryption at strong crypto cipher suites without anonymous DH. MD5 is also not accepted due to its know weaknesses. Both AES and 3DES are both enable for browser and BOT compatibility. The command "openssl ciphers -v 'HIGH:!ADH!MD5:@STRENGTH'" will show you all of the ciphers your version of OpenSSL supports in the HIGH level and not anonymous DH. Our Guide to Webserver SSL Certificates explains all of the details about ciphers and compatibility models.

ssl_prefer_server_ciphers on; just means that our server will use the ciphers specified in the "ssl_ciphers" directive over the ciphers preferred by remote clients. It is not a good security practice to ever trust remote clients.

ssl_protocols TLSv1; tells the server to only allow TLS version 1.0 or greater (TLSv1). It is highly recommended never to use SSL version 2 (SSLv2) or SSL version 3 (SSLv3) as they have vulnerabilities due to weak key strength. In order to be FIPS 140-2 compliant only TLSv1 (which stands for 1.0 or higher) can be used.

ssl_session_cache shared:SSL:1m; allows Nginx to cache the SSL session keys in its own cache structure instead of using OpenSSL slower, single threaded cache. This means Nginx can now take advantage of multiple worker_processes and separate the SSL jobs between them. The result is an impressive speed boost (2x or more) over the slower OpenSSL cache depending on your OS. The format of the line "shared:SSL:1m" is as follows: "shared" is the internal caching function, "SSL" is just an arbitrary name of this SSL cache (you can name it anything you want) and "1m"is the size of the cache (1 megabyte can hold around 4000 SSL cache sessions).

ssl_session_timeout 5m; is the cache session timeout between the client and the server set at 5 minutes or 300 seconds. When this time runs out the clients ssl session information is removed from the "ssl_session_cache". The reason this number was chosen is it also the default amount of time all client browsers will cache an ssl session. If you expect client to stay on your site longer and go to multiple pages you can always increase this default value. Depending on the client browser, they may or may not respect your ssl_session_timeout value if it larger than 5 minutes. 5 minutes seems to work perfectly fine for most sites.





How to setup a GoDaddy Turbo SSL Certificate for Nginx

  • Goto your SSL Certificate Authority company of choice (GoDaddy Turbo SSL Certificates DV sell for little as $14 a year) and purchase it. You will be asked to upload a Certificate Signing Request (CSR) by GoDaddy. Use our Guide to Webserver SSL Certificates and look for the section titled, "How to request a SSL certificate from a Certificate Authority (CA)". The explains how to use the OpenSSL binary to generate the CSR.
  • When you provide GoDaddy with your CSR, you will be offered a link to download your signed certificates at the top of the SSL page. IMPORTANT NOTE: Make sure to choose the "APACHE" certificate type when asked what format to download in; else you will not receive any usable files.
  • When you receive the certificate file it will probably be compressed in ZIP format. In the zip file will be two files: one named with your domain like "mydomain.com.crt" and one called "gd_intermediate_bundle.crt". You will need to combine both of the files for Nginx to understand them.
  • THE ORDER IS IMPORTANT: make a new file called mydomain.com_ssl.crt. Put your certificate file first into the combind crt, for example "cat mydomain.com.crt > mydomain.com_ssl.crt". Then put the gd_intermediate_bundle.crt into the combined crt file, for example "cat gd_intermediate_bundle.crt >> mydomain.com_ssl.crt". Now the mydomain.com_ssl.crt is your Nginx "ssl_certificate" file.
  • Finally, make a directory outside of the web document tree and limit access to it to root and the Nginx web daemon only. In our example we will make a directory called /ssl_keys. Both your public "key" file generated by OpenSSL (ssl_certificate_key /ssl_keys/mydomain_ssl.key) and the combined certificate "crt" file (ssl_certificate /ssl_keys/mydomain.com_ssl.crt) are copied to /ssl_keys. It is a good idea to make both file read only by the Nginx daemon. Now you can test your SSL server.




Why setup a "default blank SSL server" ?

When a client browser asks for your server they should request the hostname you registered your SSL certificate for. In our example, we registered www.example.com and example.com. If a remote client asks for anything other than these they should not be allow to get to our site. The reason being that if they did not ask for our site by name then the SSL certificate is going to be invalid. The second reason is security. If you are not asking for our site they you should not be able to get to our site.

A "default blank SSL server" is the catch all for any client not specifically asking for our site by name.

In order to allow multiple SSL certificates to be served from a single IP address we need to use virtual hosting with Server Name Indication. This means that your nginx build must report that it is supporting TLS with SNI like this:

user@machine$  nginx -V
nginx version: nginx/0.8.45
TLS SNI support enabled

If your server supports SNI you are good to go. If not, you will probably need to upgrade your version of Nginx or possibly get a newer version of OpenSSL. The example above used Nginx v0.8.45 and OpenSSL v1.0.0a for example. Now you can setup a second server block in Nginx.conf.

The following code from our "SSL only webserver" example above will tell Nginx to serve out the blank, self-signed SSL certificate for any clients not using the hostname www.example.com or example.com. This includes any scanners looking at the ip address of the server or any bad clients using false "Host:" headers.

## https .:. default blank SSL server
server {
listen 127.0.0.1:443 default;
server_name _;
ssl on;
ssl_certificate ssl_keys/default_blank.crt;
ssl_certificate_key ssl_keys/default_blank.key;
return 403;
}

We need to generate the certificate "crt" and public "key" files for Nginx. The following commands will make a self-signed certificate and key file without a pass phrase. The ssl cert is blank will not give any information to anyone looking at it.

First we need to make a new key. This will be a 4096 bit key signed using AES at 256 bits. Put in any pass phrase you want because we are going to remove it in the next step.

openssl genrsa -aes256 4096 > default_blank.key

Next, this is a dummy key we really do not care about so this command will remove the pass phrase.

openssl rsa -in default_blank.key -out default_blank.key

Third, create a certificate signing request. The only question you need to answer if the first one for Country. Put any two(2) letters in like "US" and hit enter for the rest. Make sure to keep the "hostname" entry blank too.

openssl req -new -key default_blank.key -out default_blank.csr

self sign your own certificate.

openssl x509 -req -days 1460 -in default_blank.csr -signkey default_blank.key -out default_blank.crt

Finally, copy the default_blank.crt and default_blank.key into your "ssl keys" directory so Nginx can find them.

Testing the setup

Using openssl and the s_client directive we can query the server for the proper hostname and server name using SNI. For this example we will use our real hostname calomel.org. What you are looking for in the output is the "Certificate chain" information. The real SSL cert will show multiple entries for the CA like GoDaddy. The blank SSL cert will show zero certificates in the chain and no real information. The following line will properly access the server.

openssl s_client -servername calomel.org -connect calomel.org:443

The second test should be to test a failure. Here we connect to the proper hostname calomel.org on port 443, but we ask for the wrong hostname in the SSL certificate with the "-servername" directive.

openssl s_client -servername NotTheRightHost.com -connect calomel.org:443

NOTE: there are some clients that can not use SNI enabled servers. Very OLD browsers like IE5 and Firefox v1.0 are not compatible. Strangely, some modern versions of Wget are also not compatible, but Curl and Elinks are. Modern OS's and browsers do support SNI as well as search bots like Googlebot, MSN and Yahoo. As and added bonus we find that a lot of the spammer bots do not negotiate SNI very well.





IMPORTANT NOTE: Why are we using the error code "return 444" ?

In the following sections you will notice we are using the directive "return" with an error code 444, i.e "return 444". This is a custom error code understood by the Nginx daemon to mean, "Close the connection using a tcp reset with the client without sending any headers."

Nginx understands that if you want to send the client the 444 error code (only 400-417 and 500-505 are allowed by RFC) then Nginx should just close the connection to the client. Just dropping the connection and sending an empty TCP reset packet to the client will deny scanner information about the server and _may_ confuse them.

So what is the difference in using an error 404 compared to an 444? Lets take a look at the header results using the cURL command to a server sending a 404 error. FYI: cURL will be sending the HEAD request to the server. Notice the server sends back a set of headers with the error code and some information about the server.

user@machine: curl -I http://www.somedomain.com/
HTTP/1.1 404 Not Found
Server: Nginx
Date: Mon, 10 Jan 2020 20:10:30 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=5

This is what cURL says about a Nginx server returning an error 444. The server sent nothing back to the client and closed the connection. The client did not get any useful information about the server. If you are paranoid about security or just do not want to provide and data to clients who cause errors, this is a good response for them.

user@machine: curl -I http://www.somedomain.com/
curl: (52) Empty reply from server

If you wish to change the return code to those found in the error_codes directive then your error page will be sent out instead. For example, instead of using code 444 you could send a 403 (Forbidden). For a full list of the error codes and their official definitions check out the w3.org Error Status Code Definitions.





Directive explanation and insight

Only requests to our Host are allowed : This condition is to make sure that only clients who are asking for mydomain.com or www.mydomain.com are allowed access to our server. If a client is scanning web servers then they might ask for the ip address and this is NOT allowed by our rules.

Strict Transport Security add_header : Strict Transport Security (STS or HSTS) is a proposed HTTPS response header that will require the user agent (such as a web browser) to access the website using secure connections only (such as HTTPS). The header specifies a period of time during which the user agent is not allowed to access the site insecurely. We use 2592000 seconds or 30 days.

When Strict-Transport-Security is active for a website, a complying user agent like Firefox, Internet Explorer, Opera or Chrome does the following: It automatically turns any insecure links to the website into secure links. (For instance, http://www.example.com/page/ will be modified to https://www.example.com/page/ before accessing the server.) Secondly, if the security of the connection cannot be ensured (e.g. the server's TLS certificate is self-signed), show an error message and do not allow the user to access the site despite the error. Strict-Transport-Security helps protect website users against some passive (eavesdropping) and active network attacks. A man-in-the-middle attacker will not be able to intercept any request to a website while the user's browser has Strict-Transport-Security active for that site. -Wikipedia (STS)

Only allow GET and HEAD request methods : Request Method restrictions allow you to filter on GET, HEAD, POST, SEARCH, etc. We will be limiting access to our example server to GET and HEAD requests only as we do not allow uploads or any other options due to security concerns. All other request methods will get an error defined by "return 444".

Deny certain User-Agents : You may want to list out some user-agents you do not want connecting to your server. They can be scanners, bots, spammers or any one else you find is abusing your server.

Deny certain Referers : Referer spam is more of an annoyance than a problem. A web site or bot will connect to your server with the referer field referencing their web site. The idea is that if you publish your web logs or statistics then their hostname will show up on your page. When a search bot like Google comes by it will see the link from your site to theirs and give the spammers more PageRank credit. First, never make your weblogs public. Second, block access to referer spammers with these lines.

Redirect from www to non-www : is if you prefer clients who connect to your site to instead use the non-www domain. For example, if a browser connects to www.mydomain.com they will be redirected to the URL mydomain.com with a code 301. If they then save your site location in a bookmark it will show up as the preferred non-www domain.

Stop Image and Document Hijacking : Image hijacking is when someone makes a link to your site to one of your pictures or videos, but displays it on their site as their own content. The reason this is done is to send a browser to your server to use your bandwidth and make the content look like part of the hijacker's site. This is most common as people make links to pictures and add them to a public forum or blog listing. They get to use your picture in their content and not have to use their bandwidth or server to host the file. In order to keep your bandwidth usage low you should block access to images from those clients who are not referring the content from a page on your site. Note, this function can be used for any kind on content. Just add the file types to the list. If would like more ideas on lowering bandwidth usage check out our Saving Webserver Bandwidth (Tips).

Restricted Access directory : This area is to limit access to a private or content sensitive directory. We will be limiting access to it by ip address (first check) and if that passes then ask for a password (second check). Both must match before access is granted.

access control list : This is a way you can define a directory and only allow clients coming from the specified ips to have access. Use this function to allow internal LAN clients access to the status pages or employee contact information and deny other clients. In our example we will allow the clients coming from localhost (127.0.0.1/32) and internal LAN ips 10.10.10.0/24 to access the protected "secure" directory. BTW, if you use OpenBSD's pf packet filter firewall we highly suggest enabling "synproxy" in your pf.conf for all connections to your web server. Normally when a client initiates a TCP connection to a server, PF will pass the handshake packets between the two endpoints as they arrive. PF has the ability, however, to proxy the handshake. With the handshake proxied, PF itself will complete the handshake with the client, initiate a handshake with the server, and then pass packets between the two. The benefit of this process is that no packets are sent to the server before the client completes the handshake. This eliminates the threat of spoofed TCP SYN floods affecting the server because a spoofed client connection will be unable to complete the handshake.

password protected area : If you are coming from an authorized ip address then we will ask for a username and password. If you have an area of the web site you only want authorized personnel to see then you should protect it. This set of directives will password protect the directory "/secure" and all files and directories under it. We will use the basic method which is the authors best choice to use especially for non-https sites. It will not send any of the passwords in clear text. Check "man htdigest" for details.

To make the "access_list" password file use the binary htdigest in the following form. Supply the "username" and "password" pair for access. Remember that our configuration file will look for this file at /var/www/htdocs/secure/access_list :

htpasswd -b -c access_list username password

Only allow these file types to document root : We want to restrict access to our server to clients who are actually looking for data we serve out. For example, if a client is asking for a PHP file and we do not serve that type of file then we want to deny them access.

"(^\/|\.html|\.css|\.jpg|favicon\.ico|robots\.txt|\.png)$" matches the incoming request. If a request fails to match than this service will be skipped and the client will get an error. If all services fail to match Nginx returns a generic error (next section). The example URL string specifies the file types we expect a client to want to retrieve. The dollar sign ($) says that all the strings listed must be located at the end of the request URL. This line will allow:

  • ^\/ allows the root request http://mydomain.com/ to be accepted. / is expanded into /index.html by the web server
  • \.html HTML page files
  • \.css Cascading Style Sheets
  • \.jpg JPG pictures
  • favicon\.ico is the only .ico file
  • robots\.txt is the only text file
  • \.png PNG pictures
  • $ says that each of these strings have to be located at the end of the line

Serve an empty 1x1 gif _OR_ an error 204 (No Content) for favicon.ico : Using either of these lines simply direct the Nginx daemon to serve out an empty 1x1 pixel (43 byte) favicon.ico file to the client or send back a return code 204, meaning "No content". When either option is in place you do not need a file called favicon.ico in your document root. This is perfect for anyone who sees the favicon.ico as useless and does not want to waste any bandwidth on it, but also do not want to see "file not found" errors in their logs. For more information make sure to read the section titled, "The Cursed favicon.ico" on our Webserver Optimization and Bandwidth Saving Tips page.

System Maintenance : This function will look for the file /system_maintenance.html in the document root. If the file exists then _ALL_ client requests will be redirected to this file with an error code 503 (Service Unavailable). If the file does not exist then your pages will be served as normal. The purpose is so that you can work on your site and still keep the Nginx daemon up to show helpful information to your users. The system_maintenance.html file can contain something as simple as "Site Temporarily Down. Be back up soon." It is vitally important to send clients an error code 503 in order to notify them that the page they are looking for has NOT changed, but that the site is temporally down. Google for example understands this error and will return to index your page later. If you were to redirect Google to the /system_maintenance.html page and send them a code 200 for example, Google might replace the indexing information they have with this page. Your site would then have to be completely re-indexed once you got your site back up. Definitely not what you want to happen.

All other errors get the generic error page : If the client fails the previous tests then they will receive our error_page.html. This error page will be sent out for all error codes listed (400-417, 500-505). Note the use of the internal directive? This means that an external client will not be able to access the /error_page.html file directly, only Nginx can serve this file to a client.





Starting the daemon

Make sure that the user and group Nginx is going to run as exists and can access the files in document root. Our example file will run the child daemons as the user "nginx" and the group "nginx".

Now that you have the config file installed in /etc/nginx.conf and configured you can start the daemon by hand with "/usr/local/sbin/nginx" or add the following into /etc/rc.local to start the Nginx daemon on boot.

#### Nginx start in /etc/rc.local
if [ -x /usr/local/sbin/nginx ]; then
echo -n ' Nginx'; /usr/local/sbin/nginx
fi




In Conclusion

Nginx has many more options not covered in this how to. We highly suggest taking some time to read the Nginx English Wiki if you need more web functionality. If you are happy with the options we have started out with then at this point all that is left is finding some content to serve and setup your web tree.





Strip Unnecessary White space like spaces, tabs and new line characters

How much bandwidth can you expect to save by stripping out white space? On average , one could expect to save 2% of the total size of the HTML pages written by hand or previously unoptimized. If your average page size is 100 kilobytes you could save around 2 kilobytes every time they were served. If you served a hundred thousand pages per day you could reduce your bandwidth usage by 200 megabytes per day. Not a lot, but every bit makes a difference.

This is a simple perl script called "strip_whitespace.pl". It will read in any html file and output the stripped version. Use this script to make a published copy of your html docs while keeping the human readable versions in a private directory. BTW, we use this code with the pre-compression option in Nginx to serve out pre-stripped, pre-compressed files to save on bandwidth and CPU time.

#!/usr/bin/perl -w
#
## Calomel.org -- strip_whitespace.pl
#
## PURPOSE: This program will strip out all
## whitespace from a HTML file except what is
## between the pre and /pre and tags.
#
## DEPENDANCIES: p5-HTML-Parser which you can get by CPAN or
## installing a package from your OS supporters.
#
## USAGE: ./strip_whitespace.pl < input.html > output.html
#

use HTML::Parser;
my $preserve = 0;

# Ignore any test between the /pre tags
sub process_tag
{
my ($tag, $text) = @_;
if ($tag eq 'pre') { $preserve = 1; }
elsif ($tag eq '/pre') { $preserve = 0; }
print $text;
}

# Replace all white space with a single space except what
# is between the pre tags. This includes all tabs (\t),
# returns (\r) and new line characters (\n).
sub process_default
{
my ($text) = @_;
$text =~ s/\s+/ /g unless $preserve;
print $text;
}

undef $/;
$_ = ;

my $p = HTML::Parser->new(
start_h => [\&process_tag, 'tag,text'],
end_h => [\&process_tag, 'tag,text'],
default_h => [\&process_default, 'text']
);

$p->parse($_);

## EOL




Questions?

How can I time the latency of multiple TCP and SSL handshakes ?

Curl is the tool of choice. Here we make two(2) connections to encrypted.google.com and time the responses from both the 3 way TCP handshake and the SSL negotiation of Google's 1024 bit rsa certificate key. We see that the first connection completes the 3 way TCP handshake in 32ms and the SSL handshake finishes 95ms after that. The second connection on the next line is all 0.00's because Google allows keepalive connections. So, the second request went over the same TCP connection as the first and thus saved us time. Keepalive's are quite useful when used correctly.

### bash shell
$ export URL=https://encrypted.google.com
$ curl -w "tcp: %{time_connect} ssl:%{time_appconnect} .. $URL\n" -sk -o /dev/null $URL -o /dev/null $URL
tcp: 0.032 ssl:0.095 .. https://encrypted.google.com
tcp: 0.000 ssl:0.000 .. https://encrypted.google.com

How can I best optimize Nginx for HTTPS connections ?

SSL negotiations consume a bit more CPU resources than a standard http connection. Understand that the amount of CPU used for HTTPS is not excessive, but the negotiation process does add a bit of delay (latency) to the initial connection process. On multi-processor systems you want to run several worker processes which are no less than the number of available CPU cores. By cores we mean real cores and not hyperthread Intel virtual cores.

The most CPU-intensive operation in HTTPS is the SSL handshake. There are a few ways to minimize the number of operations per client:

  • First, enable keepalive connections to send several requests via one connection.
  • Second, reuse SSL session parameters to avoid SSL handshakes for parallel and subsequent connections.
  • Third, decide how many clients will connect per second and figure out if your cipher and hardware can handle the load

Enable keepalives

Enable the "keepalive" directive to allow a remote client to send multiple queries per TCP connections. Take a look at our examples above concerning keepalive_requests and keepalive_timeout.

keepalive_requests    10;
keepalive_timeout 60 60;

Setting up SSL session cache

The sessions stored in an SSL session cache are shared between workers and configured by the ssl_session_cache directive. One megabyte of the cache contains around four thousand (4000) sessions. The default cache timeout is 5 minutes and this can be increased by using the ssl_session_timeout directive. Here is a sample of the "Option 2" configuration from above. Just like in the example Nginx is optimized for a quad core system with 10M shared session cache:

worker_processes  4;

http {
## Global SSL options
ssl_ciphers HIGH:!ADH:!MD5;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;

server {
listen 443;
server_name example.com www.example.com;
keepalive_timeout 60;

ssl on;
ssl_certificate /ssl_keys/mydomain.com_ssl.crt;
ssl_certificate_key /ssl_keys/mydomain_ssl.key;
...

How many ssl clients and how fast can we encrypt data?

You need to know how many SSL enable clients will be connecting to your servers per second and how fast you can encrypt the data flow. As we stated before the majority of time with SSL connections is taken up by the SSL handshake. The cipher you choose and the CPU in your machine are going to be the determining factors on how fast you negotiate with clients and how fast you encrypt data.

For these tests we will be using an Intel Quad core Xeon CPU L5630 @ 2.13GHz with 4gig of 1333MHz ram. The test OS is OpenBSD v4.9 (stable) and OpenSSL 1.0.0a. AES-NI (AES New Instructions) or the Advanced Encryption Standard (AES) Instruction Set can be enabled in our BIOS and is supported by our CPU. AES-NI is an extension to the x86 instruction set architecture for microprocessors from Intel and AMD with the purpose to improve the speed of applications performing encryption and decryption using the Advanced Encryption Standard (AES). The AES standard is comprised of three block ciphers, AES-128, AES-192 and AES-256, adopted from a larger collection originally published as Rijndael.

The first test will show us how fast our system can sign our ssl certificate during a handshake and how many ssl clients we can handshake with per second. A handshake is the action of the client and server opening up an encrypted connection between each other and negotiating with the site's SSL certificate. The common size of a SSL certificate is 1024, 2048 or 4096 bits. For example we sign calomel.org with a rsa 4096 bit key. So, when a client connects to our site they must negotiate with us with a rsa 4096 bit certificate.

The "sign" and "sign/s" are the values we want to examine. Make special note that the first results are for a single core and Nginx can work with multiple cores depending on your "worker_processes" directive. At rsa 4096, like what calomel.org is using, openssl specifies this machine can handshake with 44.9 clients per second per core and complete each certificate signing in 22ms (0.022). At rsa 2048 openssl can handle 300.1 SSL handshakes per second per core and sign in 0.3ms (0.003).

Note: AES-NI does NOT increase the speed of handshakes at all.

#### Intel Quad core Xeon CPU L5630 @ 2.13GHz
user@machine: openssl speed rsa
sign verify sign/s verify/s
rsa 4096 bits 0.022267s 0.000336s 44.9 2977.1 (1 core)
rsa 2048 bits 0.003332s 0.000092s 300.1 10814.8 (1 core)
rsa 1024 bits 0.000535s 0.000030s 1867.7 33674.9 (1 core)

user@machine: openssl speed -multi 4 rsa
sign verify sign/s verify/s
rsa 4096 bits 0.005498s 0.000084s 181.9 11922.5 (4 cores)
rsa 2048 bits 0.000831s 0.000023s 1203.6 43244.5 (4 cores)
rsa 1024 bits 0.000134s 0.000007s 7435.1 134482.8 (4 cores)

The second test shows how much block cipher data our system can encrypt in real time. This simulates a bulk data transfer like uploading or downloading a large file encrypted over SSL after the handshake is completed. We see that our machine and openssl can process over (145855.83k / 1024) 142 megabytes per second per core of AES 256bit, 8192 byte chunked encrypted data. This is where AES-NI can help the most. If we enable AES-NI we can almost quadruple the encryption / decryption speed as a single core can processes (402330.62k / 1024) 392 megabytes per second per core up from 142 megabytes per second per core. If we use all four(4) cores we could saturate a 10 gigabit link.

### single core AES 256bit encryption
user@machine: openssl speed -engine aesni aes-256-cbc
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256 cbc 51537.86k 53793.62k 54669.65k 143895.81k 145855.83k (single core)
aes-256-cbc 269689.64k 328777.67k 401489.65k 400522.80k 402330.62k (single core AES-NI on)

### four(4) cores AES 256bit encryption
user@machine: openssl -multi 4 -engine aesni speed aes-256-cbc
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
evp 197800.42k 212958.32k 218247.91k 574456.18k 578834.20k (4 cores)
evp 1078184.98k 1222598.23k 1574396.94k 1610181.83k 1609346.98k (4 cores AES-NI on)

SSL optimization conclusions

How do I know if my machine supports AES-NI? The new range of Intel and AMD CPU's have support for AES-NI. You should see a section in your BIOS called "AES-NI" under the CPU processor options. You can also check to see if your OS supports AES-NI. We used OpenBSD v4.9 stable and this does. You can use the command "openssl engine -t" and it should report "(aesni) Intel AES-NI engine [available]". If you see "(no-aesni)" anywhere on the line then AES-NI is NOT supported for your hardware even though the engine is available. Again, you need to make sure you enable the AES-NI option in the BIOS if available before the OS can support the option. Be aware that in order for you to take advantage of AES-NI the hardware (Intel L5630 CPU), the BIOS, the OS (OpenBSD v4.9 stable) and the program (OpenSSL 1.0.0.a) you use must all support AES-NI instructions.

What if you need more speed than a single machine can offer? In most cases it is more cost effective to buy a weaker SSL certificate (1024 bit or 2048 bit compared to 4096 bit) than it is to buy SSL hardware. Looking at the results of the test above we see our machine can sign a 4096bit key 181.9 handshakes per second using all four(4) cores. What if you need to support 500 handshakes per second? You can buy more machines or you could just use a weaker rsa key if your business plan allows it. Our tests show this machine can support 1203.6 handshakes per second when using a rsa 2048 certificate key.

You can also take a look at some hardware SSL accelerator cards. They are beyond the scope of this article. What we have seen is hardware cards add too much overhead to the process to be truly viable and they normally do not help to speed up the handshake process. The hardware cards normally just speed up the AES block cipher encryption like AES-NI did. We prefer to add more machines to our cluster as this is the best price to performance option. If you have a few moderately powered front end SSL web proxies this will be a lot less expensive to run than one huge machine. Also, if one of the small front end goes down the others can take over. If the huge machine dies then you are offline. We prefer the smaller modular design.

Can Nginx support many HTTPS domains on a single ip address ?

Yes. A solution for running several HTTPS servers on a single IP address is TLSv1.1 Server Name Indication extension (SNI, RFC3546), which allows a browser to pass a requested server name during the SSL handshake and, therefore, the server will know which certificate it should use for the connection. Support is available in all modern browsers. Note: regardless of server SNI support, older browsers always get the certificate of default server and they complain if a server name does not match a certificate's server name. Theoretically, after this step you may redirect them to an other server, but it's too late from user point of view.

In order to use SNI in Nginx, SNI must be supported in both the OpenSSL library with which the Nginx binary has been built as well as the library to which it is being dynamically linked at run time. OpenSSL supports SNI since 0.9.8f version if it was built with config option --enable-tlsext. Since OpenSSL 0.8.9j this option is enabled by default. If nginx was built with SNI support, then nginx will show this when run with the -V switch:

$ nginx -V
...
TLS SNI support enabled
...

However, if the SNI-enabled nginx is linked dynamically to an OpenSSL library without SNI support, nginx displays the warning:

nginx was built with SNI support, however, now it is linked
dynamically to an OpenSSL library which has no tlsext support,
therefore SNI is not available

When using SSL, where does Nginx get its entropy ?

Nginx uses OpenSSL's default entropy source. On Unix systems OpenSSL will try to use use /dev/urandom, /dev/random, /dev/srandom one after another. On FreeBSD /dev/urandom is symlink to /dev/random. On OpenBSD /dev/arandom is used. We highly advise using /dev/arandom On OpenBSD or FreeBSD if possible as it is extremely fast and uses high entropy. /dev/arandom uses the arc4random() method.

The arc4random() function provides a high quality 32-bit pseudo-random number very quickly. arc4random() seeds itself on a regular basis from the kernel strong random number subsystem described in random(4). On each call, an ARC4 generator is used to generate a new result. The arc4random() function uses the ARC4 cipher key stream generator, which uses 8*8 8-bit S-Boxes. The S-Boxes can be in about (2**1700) states.

The entropy source can be redefined by using the "SSLRandomSeed" directive and pointing to the new device. For example you can use the Simtec Electronics Entropy USB Key for high-quality random number generation.

For more detailed information about entropy check out our Entropy and random number generators page.

Is Nginx susceptible to the Slowloris DoS attack like Apache ?

It can be. Slowloris (slowloris.pl) holds connections open by sending partial HTTP requests. It continues to send subsequent headers at regular intervals to keep the sockets from closing. In this way webservers can be quickly tied up. In particular, servers that have threading enabled will tend to be more vulnerable, by virtue of the fact that they attempt to limit the amount of threading they'll allow.

Slowloris will wait for all the sockets to become available before it's successful at consuming them, so if it's a high traffic website, it may take a while for the site to free up it's sockets. So while you may be unable to see the website from your client, others may still be able to see it until all sockets are freed by them and consumed by Slowloris. This is because other users of the system must finish their requests before the sockets become available for Slowloris to consume.

Though it is difficult to be completely immune to this type of attack, Nginx is quite resilient. You can practice good web server security by doing the following:

  1. Limit the amount of connections and how fast those connections can be made from any ip address to your web server by use of your firewall. This can be done with PF (we have a "how to") or Iptables.
  2. Time out clients who take too long to perform any action.
  3. Drop clients immediately who send invalid data.
  4. Limit the amount or memory, cpu time and system resources the webserver can use. This is so the webserver can not take down the machine if the site is attacked.

The following Nginx directives will timeout clients who take too long to communicate their intentions to the server. The ignore_invalid_headers directive will drop any client trying to send invalid headers to the server. The explanations of each one is listed above on this page.

  • client_header_timeout 60;
  • client_body_timeout 60;
  • keepalive_timeout 60;
  • ignore_invalid_headers on;
  • send_timeout 60;

How do I setup log rotation for Nginx logs on OpenBSD?

Add the following two lines to the bottom of the /etc/newsyslog.conf file. This will rotate the logs if they are larger then 512KB and gzip the old file. We will keep a total of 4 log files.

root@machine# vi /etc/newsyslog.conf

# logfile owner:group mode count size when flags
/var/log/nginx/access.log root:wheel 644 4 512 * Z /var/run/nginx.pid
/var/log/nginx/error.log root:wheel 644 4 512 * Z /var/run/nginx.pid

Is it possible to ask Nginx to look in Memcached, but if not found to look on the local file system before passing to the back end? Would this make things more efficient?

Yes, it's possible, but it would NOT be more efficient.

Sending the file from the local file system is most efficient way if kernel VM has cached that file: then sendfile() will be a zero-copy operation: you save memory and CPU time. Working with memcached or a back end server requires:

  • copy operations to the kernel by memcached or back end systems,
  • then copy operations from the kernel by nginx,
  • then copy operations to the kernel by nginx

If memcached or a back end server are on the same computer as nginx, then all these operations involve context switches between nginx and memcached or back end.

quoted from the author, Igor Sysoev

In what circumstances would Nginx take advantage of multiple CPUs or cores?

From my experience nginx needs more CPUs in 3 cases:

  • nginx does a lot of gzip'ing
  • nginx handles many SSL connections
  • the kernel processes a lot of TCP connections of around 3,000 requests/s.

For example, one could use a quad core for a PHP back end and a dual core for the nginx proxy to handle the connections and compressing the output.

paraphrased from the author, Igor Sysoev

I am interested in setting up a SSL encrypted server (https port 443). Can your help me?

Yes. Please check out our Guide to Webserver SSL Certificates. Setting up SSL through Nginx is quite easy. What is most important is understanding how SSL encryption works and what your expectations should be.

Do you have a listing of all of the nginx directives?

The nginx wiki site has all of the config directives listed on the Nginx Wiki Configuration File Options page.

Can you suggest a good web page optimization reporting site?

We highly suggest taking a look at PageTest - Web Page Optimization and Performance Test. It is free and will analyze a page and show graphs pertaining to your sites performance and speed of delivery.

How can I test to see if my server is actually compressing/gzip'ing served files?

Look at the logs and take a look at the end of each line. In the log format directive from our example above we defined the $gzip_ratio to show if a file is compressed. If you would rather use an external site for verification then check out WhatIsMyIp Gzip Test page. Put in the full URL in question and it will display the results. Finally, you can use the utility cURL, "curl -I --compressed http://localhost". This will show you the headers returned from the server. Look for a header "Content-Encoding: gzip".

I am looking for a simpler web server. Can your help me?

If you want a dead simple web server then check out the tHttpd "how to". It is a webserver that is ready to use by executing one line and it works really well too.

No comments:

Router Packet Networking

Đây là video ngắn khá hay, mô tả đường đi của một gói tin trên Mạng Internet.