Nginx常用模块及配置（二）

Nginx常用模块及配置没写完，来继续。

本篇也是这个实验的延续。

上个实验简单实现了负载均衡，使用的默认的轮询算法。

一、负载均衡

一般指的七层负载均衡，虽然 Nginx 也能实现四层，不过一般常用于 LVS 去实现。

官方文档

负载均衡算法 说明
rr轮询 round robin 轮询，默认循环访问
wrr 加权轮询，实验 weight 关键词
ip_hash ip哈希，同一个ip地址会匹配到同一台后端服务器，默认使用IP地址的前24位
xxx_hash url_hash 访问同一路径会匹配到同一台后端服务器
least_conn 最小连接数，lc算法，也可同时配置 weight，会分流到最小连接数的后端服务器

1.1 rr轮询

1
vim /etc/nginx/conf.d/admin.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
upstream adminsPool{
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}

1.2 wrr轮询

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
upstream adminsPool{
    server 192.168.10.101:80 weight=1;
    server 192.168.10.102:80 weight=6;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}

1.3 批量验证脚本

需提前做好解析

1
1.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash

URL=${1:-http://admin.bravexist.cn/info.php}
COUNT=${2:-10}

c200=0
c404=0

for ((i=1; i<=COUNT; i++)); do
  code=$(curl -s -o /dev/null -w "%{http_code}" "$URL")

  if [[ "$code" == "200" ]]; then
    ((c200++))
  elif [[ "$code" == "404" ]]; then
    ((c404++))
  fi
done

echo "URL: $URL"
echo "总请求次数: $COUNT"
echo "200 次数: $c200"
echo "404 次数: $c404"
1
touch 1.bat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@echo off
set URL=http://admin.bravexist.cn/info.php
set COUNT=100

set OK=0
set NOTFOUND=0

for /L %%i in (1,1,%COUNT%) do (
    for /f "delims=" %%c in ('
        curl -s -o NUL -w "%%{http_code}" %URL%
    ') do (
        if "%%c"=="200" set /a OK+=1
        if "%%c"=="404" set /a NOTFOUND+=1
    )
)

echo URL: %URL%
echo 总请求次数: %COUNT%
echo 200 次数: %OK%
echo 404 次数: %NOTFOUND%

1.4 IP哈希

因为是内网，确实只能请求到一台机器上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
upstream adminsPool{
    ip_hash;
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}

1.5 URL哈希

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
upstream adminsPool{
    hash $request_uri;
    # 或 
    # hash $uri;
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}

1.6 最小连接数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
upstream adminsPool{
	least_conn;
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}

1.7 other

自己探索，放一下生产环境的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
upstream adminsPool{
	
    server 192.168.10.101:80 weight=1 max_fails=3 fail_timeout=30s;
    server 192.168.10.102:80 weight=1 max_fails=3 fail_timeout=30s;
    server 192.168.10.103:80 backup;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
}
  • max_fails 达到失败次数就认为宕机
  • fail_timeout 达到失败时长后就再次尝试
  • backup 只有其它机器都宕机才会启用

1.8 四层负载均衡

参数此处

二、Tengine编译安装及平滑升级

主动检测后端服务器的模块，需要自己编译第三方模块，顺带验证平滑升级。

Tengine 是淘宝对 Nginx 的二次开发。默认附带一些第三方模块。

2.1 编译

使用额外的一台机器，单独进行编译安装。

  1. 安装依赖
1
yum install -y pcre pcre-devel openssl openssl-devel
  1. 下载
1
wget https://tengine.taobao.org/download/tengine-3.1.0.tar.gz
  1. 解压
1
tar xf tengine-3.1.0.tar.gz
  1. 编译前的检查，缺少依赖会报错
1
./configure

​ 为了平滑升级的顺利，添加之前 nginx 编译的模块及安装路径。

1
nginx -V
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
./configure --prefix=/etc/nginx \
        --sbin-path=/usr/sbin/nginx \
        --modules-path=/usr/lib64/nginx/modules \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --pid-path=/var/run/nginx.pid \
        --lock-path=/var/run/nginx.lock \
        --http-client-body-temp-path=/var/cache/nginx/client_temp \
        --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
        --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
        --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
        --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
        --user=nginx \
        --group=nginx \
        --with-compat \
        --with-file-aio \
        --with-threads \
        --with-http_addition_module \
        --with-http_auth_request_module \
        --with-http_dav_module \
        --with-http_flv_module \
        --with-http_gunzip_module \
        --with-http_gzip_static_module \
        --with-http_mp4_module \
        --with-http_random_index_module \
        --with-http_realip_module \
        --with-http_secure_link_module \
        --with-http_slice_module \
        --with-http_ssl_module \
        --with-http_stub_status_module \
        --with-http_sub_module \
        --with-http_v2_module \
        --with-mail \
        --with-mail_ssl_module \
        --with-stream \
        --with-stream_realip_module \
        --with-stream_ssl_module \
        --with-stream_ssl_preread_module \
        --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' \
        --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' \
        --add-module=modules/ngx_http_upstream_check_module \
        --add-module=modules/ngx_http_upstream_session_sticky_module
  1. 编译
1
make -j 2
  1. 安装（可选，自行判断）
1
make install
  1. 查看编译产物下的 nginx
1
ls -l ./objs
  1. 验证
1
./objs/nginx -v

2.2 平滑升级

  1. 备份旧的二进制文件，默认 /usr/sbin 和 /sbin 目录是软链接。
1
mv /usr/sbin/nginx /usr/sbin/nginx.old
  1. 移动编译后的 nginx 到目标文件
1
scp ./objs/nginx root@192.168.10.103:/usr/sbin/nginx
  1. 验证替换成功，并且兼容旧配置文件
1
nginx -V
1
nginx -t
  1. 查看旧进程
1
ps -ef |grep nginx
1
cat /var/run/nginx.pid
  1. 发送 USER2 信号
1
kill -USR2 `cat /var/run/nginx.pid`
  1. 验证是否启动新进程
1
ps -ef |grep nginx
  1. 逐步关闭老进程

​ 让老 Worker 停止接收新连接

1
kill -WINCH `cat /var/run/nginx.pid.oldbin`

​ 老 Master 停止退出

1
kill -QUIT `cat /var/run/nginx.pid.oldbin`
  1. 再次验证
1
ps -ef |grep nginx

三、 upstream_check_module 模块

此模块可以对上游服务器进行主动检查。详见官方文档。

3.1 官网案例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
http {
    upstream cluster1 {
        # simple round-robin
        server 192.168.0.1:80;
        server 192.168.0.2:80;

        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_http_send "HEAD / HTTP/1.0\r\n\r\n";
        check_http_expect_alive http_2xx http_3xx;
    }

    upstream cluster2 {
        # simple round-robin
        server 192.168.0.3:80;
        server 192.168.0.4:80;

        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_keepalive_requests 100;
        check_http_send "HEAD / HTTP/1.1\r\nConnection: keep-alive\r\n\r\n";
        check_http_expect_alive http_2xx http_3xx;
    }

    server {
        listen 80;

        location /1 {
            proxy_pass http://cluster1;
        }

        location /2 {
            proxy_pass http://cluster2;
        }

        location /status {
            check_status;

            access_log   off;
            allow SOME.IP.ADD.RESS;
            deny all;
        }
    }
}

upstream cluster1

  • 3000，每 3000ms 检查一次

  • rise=2 连续两次成功才认为可用

  • fall=5 连续五次失败才认为不可用

  • timeout=1000 1 秒超时

  • type=http HTTP 检查

  • HEAD / HTTP/1.0 是HTTP 1.0版本的请求

upstream cluster2

  • 基本同 cluster1，但启用了 keepalive
    • check_keepalive_requests 100 → 每个健康检查连接可以复用 100 次
  • 使用 HTTP/1.1 和 Connection: keep-alive，减少 TCP 建立开销

3.2 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
upstream adminsPool{
	
    server 192.168.10.101:80;
    server 192.168.10.102:80;
    check interval=3000 rise=2 fall=5 timeout=1000 type=http;
    # check_http_send "HEAD / HTTP/1.0\r\n\r\n";
    # 默认使用的ip请求，不携带Host，此处加以改造
    check_http_send "HEAD / HTTP/1.0\r\nHost:admin.bravexist.cn\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
    
    location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }

    location /status {
        check_status;
        access_log   off;
        allow 192.168.10.1;
        deny all;
    }
}

3.3 验证

http://admin.bravexist.cn/status

1
http://admin.bravexist.cn/status

四、rewrite 功能

指令 说明
return 实现对url的改写，一般与ngx变量一起使用，返回指定的状态码
rewrite 实现对url的改写，使用正则匹配uri，进行改写，还有各种标记
set 创建或修改ngx的变量
if 判断，一般与ngx变量一起使用

4.1 return

格式如下，可以放到 server、location、if 等

  • return code URL;
  • return code;
1
vim /etc/nginx/conf.d/admin.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
upstream adminsPool{
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;

    location / {
        proxy_pass http://adminsPool;

        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
	
    location /admin001 {
        return 666;
    }
    location /admin002 {
        return 301 https://bravexist.cn;
    }
}

验证

1
2
curl -v http://admin.bravexist.cn/admin001
curl -v http://admin.bravexist.cn/admin002

4.2 if

Ngx变量

1
vim /etc/nginx/conf.d/admin.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
upstream adminsPool{
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;

    location / {
        proxy_pass http://adminsPool;

        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
    if ( $request_method !~ "GET|POST" ) {
        return 301 https://bravexist.cn;
    }
}
1
nginx -t && nginx -s reload
1
2
curl  http://admin.bravexist.cn -I -X GET
curl  http://admin.bravexist.cn -I -X HEAD

4.3 set

1
vi /etc/nginx/conf.d/admin.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
upstream adminsPool{
    server 192.168.10.101:80;
    server 192.168.10.102:80;
}
server{
    listen 80;
    server_name admin.bravexist.cn;
	set $flag 1;
    # set $flag 0;
    location / {
        proxy_pass http://adminsPool;

        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    }
    location /testset {
        set $name "qiankong";
        return 200 "hello $name";
    }
    location /testset002 {
        set $name $http_host$request_uri;
        return 200 "hello $name";
    }
    if ( $flag = 0 ){
        return 503;
    }
}
1
nginx -t && nginx -s reload
1
2
curl  http://admin.bravexist.cn/testset
curl  http://admin.bravexist.cn/testset002

4.4 rewrite

重写uri，替换原来的uri。

1
2
3
4
5
6
server{
    listen 80;
    server_name admin.bravexist.cn;
	rewrite ^(.*)$    http://www.baidu.com$1 ;
    # rewrite ^(.*)$    http://www.baidu.com$1 permanent;
}
  • permanent 永久重定向
  • break 会继续匹配后续的 location
  • last 不会继续匹配后续的 location
  • redirect 临时重定向

五、HTTPS

5.1 配置

本质就是配置证书和私钥，80端口重定向到443端口，可以设置SSL版本、加密算法，以及其它的一些SSL参数。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
server {
	listen 80;
	server_name ssl.bravexist.cn;
	return 301 https://ssl.bravexist.cn$request_uri;
}
server {
	listen 443 ssl;
	server_name ssl.oldboylinux.cn;
	root /www/wwwroot/ssl;
	
    #ssl key
	ssl_certificate /etc/nginx/ssl_keys/ssl.bravexist.cn.pem;
	ssl_certificate_key /etc/nginx/ssl_keys/ssl.bravexist.cn.key;
	
    ssl_session_cache shared:le_nginx_SSL:10m;
    ssl_session_timeout 1440m;
    ssl_session_tickets off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

    location / {
		index index.html;
	}
}

如果涉及到负载均衡，会涉及到请求上游服务器时，是否使用HTTPS协议。

  • 全流程HTTPS需要同步配置SSL证书，会遇到更少的问题。
  • 请求上有服务器时使用HTTP的话，需要按照实际情况设置一些参数，告诉上游服务器，客户端使用HTTPS协议。
1
2
3
4
5
6
7
8
9
10
11
12
location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass unix:/run/php-fpm/www.sock;
        fastcgi_index index.php;
    	
    	# SSL
    	fastcgi_param HTTPS               on;
    	fastcgi_param HTTP_X_FORWARDED_PROTO https;
    	fastcgi_param HTTP_X_FORWARDED_SSL   on;
        
    	fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
1
2
3
4
5
6
7
8
9
10
11
location / {
        proxy_pass http://adminsPool;
        
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;

        # 协议相关（重点）
		proxy_set_header X-Forwarded-Proto $scheme;

    }

5.2 自签名证书

5.2.1 生成自签名证书

法一，同时生成证书和私钥

1
2
3
4
5
6
openssl req -x509 -newkey rsa:2048 \
  -keyout server.key \
  -out server.crt \
  -days 36500 \
  -sha256 \
  -nodes

法二，

  1. 生成私钥
1
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048
  1. 生成证书
1
2
3
4
5
openssl req -new -x509 \
  -key server.key \
  -out server.crt \
  -days 36500 \
  -sha256

5.2.2 查看证书信息

1
openssl x509 -in server.crt -noout -text

5.3 证书自动化

参考HTTPS证书的签发

六、高可用

参考Keepalived实现Nginx高可用