本文是我給公司內(nèi)部寫的一個(gè)簡單的配置文檔,文中只有配置步驟,省掉了原理說明部分。
polygun2000原創(chuàng),轉(zhuǎn)載請注明: 來源于polygun2000博客
http://blog.sina.com.cn/polygun2000
一、功能需求
1.四層負(fù)載均衡(TCP)和七層負(fù)載均衡(HTTP)
2.會話保持
二、系統(tǒng)結(jié)構(gòu)
haproxy: http://haproxy.1wt.eu
1.基于 TCP 和 HTTP 協(xié)議的高效能負(fù)載均衡器(不同于nginx,haproxy本身不具有web server功能)。
2.基于GPL協(xié)議,開源軟件。
3.高效,穩(wěn)定,安全性高,適合重負(fù)載使用,支持10GE網(wǎng)卡。
4.負(fù)載均衡算法靈活: 輪詢,靜態(tài)輪詢,最小連接數(shù),源地址hash,基于url等。
5.支持透明代理,限速等高級功能。
tproxy: http://www.balabit.com/support/community/products/tproxy
1.支持透明代理的內(nèi)核補(bǔ)丁,自2.6.28以后已經(jīng)進(jìn)入主線內(nèi)核。
2.結(jié)合haproxy可以使用戶IP地址透傳給后端服務(wù)器。
keepalived: http://www.keepalived.org
1.用來防止路由器出現(xiàn)單點(diǎn)故障的熱備份軟件,最早用于與LVS結(jié)合。
2.使用VRRP協(xié)議。
四、配置過程簡述
五、具體配置步驟
1.環(huán)境準(zhǔn)備
硬件選擇: E5-2600CPU+Intel服務(wù)器網(wǎng)卡
操作系統(tǒng): 最小化安裝CentOS 6.3 x86_64
a.關(guān)閉網(wǎng)卡中斷調(diào)節(jié)
[root@ modprobe.d]# vi /etc/modprobe.d/intel-nic.conf
options igb
InterruptThrottleRate=0,0,0,0
或者
options ixgbe
InterruptThrottleRate=0,0
b.設(shè)置網(wǎng)卡中斷CPU親和
set_irq_affinity.sh腳本包含在Intel官方的ixgbe驅(qū)動中,下載地址:
https://downloadcenter.intel.com/download/14687/Network-Adapter-Driver-for-PCI-E-10-Gigabit-Network-Connections-under-Linux-
安裝163,epel源
[root@haproxy ~]#yum install wget
[root@haproxy ~]#wget http://mirrors.163.com/.help/CentOS6-Base-163.repo
[root@haproxy ~]#wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
[root@haproxy ~]#mv CentOS6-Base-163.repo /etc/yum.repos.d/CentOS-Base.repo
[root@haproxy ~]#rpm -ivhepel-release-6-8.noarch.rpm
[root@haproxy ~]#yum update
2.編譯安裝pcre
[root@haproxy ~]#yum install gcc gcc-c++ make zlib-devel bzip2-devel
[root@haproxy ~]#wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.32.tar.bz2
[root@haproxy ~]#tar xvjf pcre-8.32.tar.bz2
[root@haproxy ~]#./configure --prefix=/usr \
--docdir=/usr/share/doc/pcre-8.32 \
--enable-utf --enable-unicode-properties \
--enable-pcregrep-libz --enable-pcregrep-libbz2
[root@haproxy ~]#make
[root@haproxy ~]#make check
[root@haproxy ~]#make install
3.編譯安裝haproxy
[root@haproxy ~]#yum install openssl-devel
[root@haproxy ~]#wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev17.tar.gz
[root@haproxy ~]#tar xvzfhaproxy-1.5-dev17.tar.gz
[root@haproxy ~]#cd haproxy-1.5-dev17
[root@haproxy ~]#make TARGET=linux26 USE_STATIC_PCRE=1 \
USE_REGPARM=1 USE_LINUX_TPROXY=1 USE_OPENSSL=1 USE_ZLIB=1 ARCH=x86_64
[root@haproxy ~]#make install
4.創(chuàng)建haproxy啟動腳本
直接下載連接: http://mattiasgeniar.be/downloads/haproxy/haproxy.init
[root@haproxy ~]#vi /etc/init.d/haproxy
#----------------------------
#!/bin/sh
#
# custom haproxy init.d script, by Mattias Geniar
#
# haproxy starting and stopping the haproxy load balancer
#
# chkconfig: 345 55 45
# description: haproxy is a TCP loadbalancer
# probe: true
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
[ -f /usr/local/sbin/haproxy ] || exit 0
[ -f /etc/haproxy/haproxy.conf ] || exit 0
# Define our actions
checkconfig() {
# Check the config file for errors
/usr/local/sbin/haproxy -c -q -f /etc/haproxy/haproxy.conf
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
# We're OK!
return 0
}
start() {
# Check config
/usr/local/sbin/haproxy -c -q -f /etc/haproxy/haproxy.conf
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
echo -n "Starting HAProxy: "
daemon /usr/local/sbin/haproxy -D -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/haproxy
return $RETVAL
}
stop() {
echo -n "Shutting down HAProxy: "
killproc haproxy -USR1
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/haproxy
[ $RETVAL -eq 0 ] && rm -f /var/run/haproxy.pid
return $RETVAL
}
restart() {
/usr/local/sbin/haproxy -c -q -f /etc/haproxy/haproxy.conf
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
stop
start
}
check() {
/usr/local/sbin/haproxy -c -q -V -f /etc/haproxy/haproxy.conf
}
rhstatus() {
status haproxy
}
reload() {
/usr/local/sbin/haproxy -c -q -f /etc/haproxy/haproxy.conf
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
echo -n "Reloading HAProxy config: "
/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
success $"Reloading HAProxy config: "
echo
}
# Possible parameters
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart)
restart
;;
reload)
reload
;;
checkconfig)
check
;;
*)
echo "Usage: haproxy {start|stop|status|restart|reload|checkconfig}"
exit 1
esac
exit 0
#----------------------------
[root@haproxy ~]#chmod +x /etc/init.d/haproxy
設(shè)置開機(jī)啟動haproxy服務(wù)
[root@haproxy ~]#chkconfig --add haproxy
[root@haproxy ~]#chkconfig haproxy on
5.配置haproxy
創(chuàng)建chroot目錄,確保該目錄為空,且其賬號不可訪問。
[root@haproxy ~]#mkdir /var/haproxy
[root@haproxy ~]#chmod o= /var/haproxy
創(chuàng)建haproxy配置文件
[root@haproxy ~]#mkdir /etc/haproxy
[root@haproxy ~]#vi /etc/haproxy/haproxy.conf
global段配置
#全局配置
global
maxconn 32768 # Max simultaneous connections from an upstream server
spread-checks 5 # Distribute health checks with some randomness
chroot /var/haproxy
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#debug # Uncomment for verbose logging
defaults段配置
#默認(rèn)配置,應(yīng)用于所有下邊的服務(wù)
defaults
log global
mode http
balance roundrobin
retries 3
option abortonclose # abort request if client closes output channel while waiting
option httpclose # add "Connection:close" header if it is missing
option forwardfor # insert x-forwarded-for header so that app servers can see both proxy and client IPs
option redispatch # any server can handle any session
option httplog
option dontlognull
timeout http-request 5s #aginst Slowloris attack
timeout client 60s
timeout connect 9s
timeout server 30s
timeout check 5s
stats enable
errorfile 503 /etc/haproxy/errors/503.http
stat監(jiān)控配置
#配置haproxy的狀態(tài)監(jiān)控
listen stats
bind 192.168.10.132:8888
stats uri /
stats realm Haproxy\ Statistics
stats auth hadmin:yhXV2WAbybXd1euzEXbe
stats refresh 20
log配置
1.配置rsyslog以接收haproxy日志
[root@haproxy ~]#vi /etc/rsyslog.d/haproxy.conf
# Custom log facilities for haproxy
local0.* -/var/log/haproxy0a.log
local1.* -/var/log/haproxy1a.log
$ModLoad imudp
# load the imudp module for rsyslog
# provides UDP syslog reception
# start UDP server on this port, "*" means all addresses
$UDPServerRun 514
# local IP address (or name) the UDP listens should bind to
$UDPServerAddress 127.0.0.1
[root@haproxy ~]#/etc/init.d/rsyslog restart
注釋:
/var/log/haproxy0a.log前邊的"-"減號意味著取消日志同步寫入。
這可以優(yōu)化一下磁盤寫入,尤其是在非常繁忙的系統(tǒng)中。
不過如果突然斷電,可能會損失一些未寫入硬盤的日志。
2.配置logrotate
[root@haproxy ~]#vi /etc/logrotate.d/haproxy
/var/log/haproxy*.log
{
daily
rotate 4
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/etc/init.d/haproxy reload >/dev/null
endscript
}
注釋:
如果站點(diǎn)數(shù)量較多,可能會希望將不同站點(diǎn)的日志分開,可以看看后邊的"參考文檔E"。
http應(yīng)用配置
listen VIP_64.4.2.111
bind 64.4.2.111:80
cookie SERVERID insert indirect nocache
server s31 192.168.10.31:80 check cookie s1
server s32 192.168.10.32:80 check cookie s2
tcp應(yīng)用配置
listen VIP_64.4.2.118
bind 64.4.2.118:22186
mode tcp
option tcplog
server s41 192.168.10.41:22186 check
server s42 192.168.10.42:22186 check
會話保持配置
#需要做會話保持的tcp配置,采用源地址hash
listen VIP_64.4.2.109
bind 64.4.2.109:1235
balance source
option tcplog
hash-type consistent # optional
server s11 192.168.10.11:1235 check
server s12 192.168.10.12:1235 check
#需要做會話保持的http配置
listen VIP_64.4.2.111
bind 64.4.2.111:80
cookie SERVERID insert indirect nocache
server s31 192.168.10.31:80 check cookie s1
server s32 192.168.10.32:80 check cookie s2
源地址透傳配置
#需要查看用戶真實(shí)IP的配置
listen VIP_64.4.2.118
bind 64.4.2.118:22186
mode tcp
option tcplog
source 0.0.0.0 usesrc clientip
server s41 192.168.10.41:22186 check
server s42 192.168.10.42:22186 check
為TPROXY設(shè)置iptables規(guī)則
[root@haproxy ~]#/sbin/iptables -t mangle -N DIVERT
[root@haproxy ~]#/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
[root@haproxy ~]#/sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
[root@haproxy ~]#/sbin/iptables -t mangle -A DIVERT -j ACCEPT
[root@haproxy ~]#/sbin/ip rule add fwmark 1 lookup 100
[root@haproxy ~]#/sbin/ip route add local 0.0.0.0/0 dev lo table 100
#給tproxy后端做NAT
[root@haproxy ~]#/sbin/iptables -t nat -A POSTROUTING -s backend's_ip -o eht0 -j MASQUERADE
在后端服務(wù)器上設(shè)置haproxy為默認(rèn)網(wǎng)關(guān)
[root@backend ~]# ip route add default via haproxy_lanip
5.相關(guān)內(nèi)核參數(shù)調(diào)整
[root@haproxy ~]# vi /etc/sysctl.conf
#允許ip轉(zhuǎn)發(fā)
net.ipv4.ip_forward = 1
#設(shè)置松散逆向路徑過濾
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.eth0.rp_filter = 0
#允許ICMP重定向
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.default.send_redirects = 1
#發(fā)送到一個(gè)監(jiān)聽的socket上的最大已完成連接隊(duì)列長度
#三次握手已經(jīng)完成,但還未被應(yīng)用層接收(accept),但也處于ESTABLISHED狀態(tài)
#隊(duì)列長度由listen的backlog參數(shù)和內(nèi)核的 net.core.somaxconn 參數(shù)共同決定
#當(dāng)這個(gè)隊(duì)列滿了之后,不管未完成連接隊(duì)列是否已滿,是否啟用syncookie,都不在接收新的SYN請求.
net.core.somaxconn = 32768
#允許綁定到非本地地址,用于keepalived
net.ipv4.ip_nonlocal_bind = 1
#增加可用的端口范圍
net.ipv4.ip_local_port_range = 1024 65023
#防攻擊使用,如無必要一定要設(shè)置成0
net.ipv4.tcp_abort_on_overflow = 0
#如果套接字由本端要求關(guān)閉,這個(gè)參數(shù)決定了它保持在FIN-WAIT-2狀態(tài)的時(shí)間,缺省值是60秒。
#減小這個(gè)值,可以使TCP/IP更快的釋放連接,騰出更多資源給新連接。推薦15-30秒。
net.ipv4.tcp_fin_timeout = 10
#最后一個(gè)數(shù)據(jù)包發(fā)送完成和第一個(gè)keepalive包被檢測到之間的時(shí)間間隔
#表示當(dāng)keepalive起用的時(shí)候,TCP發(fā)送keepalive消息的頻度,缺省是2小時(shí)。
net.ipv4.tcp_keepalive_time = 300
#系統(tǒng)所能處理不屬于任何進(jìn)程的TCP sockets最大數(shù)量。
#假如超過這個(gè)數(shù)量,那么不屬于任何進(jìn)程的連接會被立即reset,并同時(shí)顯示警告信息。
net.ipv4.tcp_max_orphans = 262144
#backlog隊(duì)列是一個(gè)大的內(nèi)存結(jié)構(gòu),用來處理收到的帶有SYN標(biāo)記的數(shù)據(jù)包,直到三次握手完成。
#這個(gè)參數(shù)控制了同一時(shí)間內(nèi)操作系統(tǒng)可以處理多少個(gè)半開連接,當(dāng)連接數(shù)達(dá)到這個(gè)數(shù)值的設(shè)定后,系統(tǒng)會丟棄隨后的請求。
net.ipv4.tcp_max_syn_backlog = 16384
#表示系統(tǒng)同時(shí)保持TIME_WAIT套接字的最大數(shù)量,如果超過這個(gè)數(shù)字,TIME_WAIT套接字將立刻被清除并打印警告信息。
net.ipv4.tcp_max_tw_buckets = 262144
#對于遠(yuǎn)端的連接請求SYN,內(nèi)核會發(fā)送SYN + ACK數(shù)據(jù)報(bào),以確認(rèn)收到上一個(gè) SYN連接請求包。
#這是所謂的三次握手( threeway handshake)機(jī)制的第二個(gè)步驟。這里決定內(nèi)核在放棄連接之前所送出的 #SYN+ACK數(shù)目。如果你的網(wǎng)站SYN_RECV狀態(tài)確實(shí)挺多,為了避免syn攻擊,那么可以調(diào)節(jié)重發(fā)的次數(shù)。
net.ipv4.tcp_synack_retries = 3
#開啟/關(guān)閉SYN Cookies
#當(dāng)啟動SYN Cookie時(shí),主機(jī)在發(fā)送 SYN/ACK 確認(rèn)封包前,會要求 Client 端在短時(shí)間內(nèi)回復(fù)一個(gè)序號
#這個(gè)序號包含許多原本 SYN 封包內(nèi)的信息,包括 IP、port 等。
#若 Client 端可以回復(fù)正確的序號,那么主機(jī)就確定該封包為可信的,因此會發(fā)送 SYN/ACK 封包,否則就不理會此一封包。
#這個(gè)參數(shù)不會提高性能,而且違背TCP協(xié)議,如果不是遭到SYN Flood攻擊,不要打開。
net.ipv4.tcp_syncookies = 0
#根據(jù)RFC1323,會向TCP包頭中插入12byte,2.6內(nèi)核的Linux默認(rèn)是打開的,某些情況下timestamp數(shù)值有可能溢出造成TCP超時(shí)
#建議關(guān)閉。
net.ipv4.tcp_timestamps = 0
#開啟TCP連接中TIME-WAIT sockets的快速回收
net.ipv4.tcp_tw_recycle = 1
#開啟重用,允許將TIME-WAIT sockets重新用于新的TCP連接
net.ipv4.tcp_tw_reuse = 1
#如果TCP窗口大小超過65536,需要此選項(xiàng)打開大TCP窗口支持。
net.ipv4.tcp_window_scaling=1
#決定TCP協(xié)議棧如何使用內(nèi)存,單位是內(nèi)存分頁,而不是字節(jié)。每個(gè)內(nèi)存分頁一般為4K。
#當(dāng)超過第二個(gè)值時(shí),TCP進(jìn)入pressure模式,此時(shí)TCP嘗試穩(wěn)定其內(nèi)存的使用,
#當(dāng)小于第一個(gè)值時(shí),就退出pressure模式,TCP不會考慮釋放內(nèi)存。
#當(dāng)內(nèi)存占用超過第三個(gè)值時(shí),TCP就拒絕分配socket了,查看dmesg,會打出很多的日志“TCP: too many of orphaned sockets”。
#如果不是非常必要,一般不要?jiǎng)酉到y(tǒng)默認(rèn)的值,默認(rèn)值一般來說夠用了
net.ipv4.tcp_mem = "786432 2097152 3145728"
#TCP流中重排序的數(shù)據(jù)包最大數(shù)量
net.ipv4.tcp_reordering = 3
#系統(tǒng)auto-tuning時(shí),每個(gè)socket使用的內(nèi)存。分別是最小,缺省,最大TCP接收窗口的內(nèi)存大小,單位byte
#如果設(shè)置net.core.rmem_default,則該值會覆蓋缺省值
#如果設(shè)置net.core.rmem_max,則該值會覆蓋最大值
net.ipv4.tcp_rmem = "4096 87380 16777216"
6.keepalived配置
安裝keepalived
[root@haproxy ~]#yum install keepalived
配置keepalived
[root@haproxy ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs { # global_defs全局配置標(biāo)識,說明這個(gè)區(qū)域{}是全局配置
notification_email { # 發(fā)送email通知,以及email發(fā)送給哪些郵件地址,郵件地址可以多個(gè),每行一個(gè)。
admin@demo.com
}
notification_email_from admin@demo.com # 發(fā)送通知郵件時(shí)郵件源地址是誰
smtp_connect_timeout 3 # smtp連接超時(shí)時(shí)間
smtp_server 127.0.0.1 # 發(fā)送email時(shí)使用的smtp服務(wù)器地址
router_id haproxy_101 # 機(jī)器標(biāo)識,從節(jié)點(diǎn)為haproxy_102
}
vrrp_script chk_haproxy { # 定義腳本名字
script "killall -0 haproxy"
interval 2 # 腳本執(zhí)行間隔2s
weight 10 # 腳本結(jié)果導(dǎo)致的優(yōu)先級變更:10表示優(yōu)先級+10;-10則表示優(yōu)先級-10
fall 2 #
require 2 failures for KO
rise 2 # require 2 successes
for OK
}
vrrp_instance VI_1 { # vrrp實(shí)例名稱
interface eth1 # 實(shí)例綁定的網(wǎng)卡,因?yàn)樵谂渲锰摂MIP的時(shí)候必須是在已有的網(wǎng)卡上添加的
state MASTER # 從節(jié)點(diǎn)則此此處為BACKUP ,需要大寫這些單詞
priority 101 # 設(shè)置本節(jié)點(diǎn)的優(yōu)先級,數(shù)值愈大,優(yōu)先級越高,優(yōu)先級高的為master
virtual_router_id 50 # 主、備機(jī)的virtual_router_id必須相同!!
garp_master_delay 1 # 主從切換時(shí)間,單位為秒。
authentication { # 設(shè)置認(rèn)證,同一vrrp實(shí)例MASTER與BACKUP 使用相同的密碼才能正常通信。
auth_type PASS # 認(rèn)證方式,可以是PASS或AH兩種認(rèn)證方式
auth_pass U5vXgwcveTuDt66MxJa7 # 認(rèn)證密碼
}
virtual_ipaddress { # 這里設(shè)置的就是VIP,也就是用工作的虛擬IP地址,VIP最多20個(gè)
64.4.2.110/24 dev eth0
}
virtual_ipaddress_excluded { # 超過20個(gè)VIP可以添加在virtual_ipaddress_excluded中,這些VIP不需要發(fā)送檢測包
64.4.2.111/24 dev eth0
64.4.2.112/24 dev eth0
202.113.58.7/24 dev eth1
}
track_interface { # 跟蹤接口,設(shè)置額外的監(jiān)控,里面任意一塊網(wǎng)卡出現(xiàn)問題,都會進(jìn)入故障(FAULT)狀態(tài)
eth0
eth1
}
track_script { # 引用vrrp_script,有點(diǎn)類似腳本里面的函數(shù)引用一樣,先定義,后引用函數(shù)名
chk_haproxy # 調(diào)用腳本必須放在virtual_ipaddress之后
}
#狀態(tài)通知
notify_master /etc/keepalived/scripts/be_master.sh # 當(dāng)進(jìn)入Master狀態(tài)時(shí)會呼叫notify_master
notify_backup /etc/keepalived/scripts/be_backup.sh # 當(dāng)進(jìn)入Backup狀態(tài)時(shí)會呼叫notify_backup
notify_fault /etc/keepalived/scripts/be_fault.sh # 當(dāng)發(fā)現(xiàn)異常情況時(shí)進(jìn)入Fault狀態(tài)呼叫notify_fault
notify_stop /etc/keepalived/scripts/be_stop.sh # 當(dāng)Keepalived程序終止時(shí)則呼叫notify_stop
}
確認(rèn)keepalived工作正常
[root@haproxy ~]# tcpdump -v -i eth0 host 224.0.0.18
tcpdump: listening on eth0, link-type EN10MB (Ethernet),
capture size 96 bytes
16:54:01.743275 IP (tos 0x0, ttl 255, id 451, offset 0, flags
[none], proto: VRRP (112), length: 96) 10.10.28.5 > 224.0.0.18:
VRRPv2, Advertisement, vrid 51, prio 103, authtype simple, intvl
1s, length 76, addrs(15): 123.12.15.2,123.12.15.3[|vrrp]
16:54:02.744241 IP (tos 0x0, ttl 255, id 452, offset 0, flags
[none], proto: VRRP (112), length: 96) 10.10.28.5 > 224.0.0.18:
VRRPv2, Advertisement, vrid 51, prio 103, authtype simple, intvl
1s, length 76, addrs(15): 123.12.15.2,123.12.15.3[|vrrp]
六、進(jìn)階應(yīng)用
1.限制單個(gè)IP的并發(fā)連接數(shù)
frontend ft_web
bind 0.0.0.0:8080
# Table definition
stick-table type ip size 100k expire 30s store conn_cur
# Allow clean known IPs to bypass the filter
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
2.限制單個(gè)IP建立連接的頻率
frontend ft_web
bind 0.0.0.0:8080
# Table definition
stick-table type ip size 100k expire 30s store conn_rate(3s)
# Allow clean known IPs to bypass the filter
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_rate ge 10 }
tcp-request connection track-sc1 src
3.限制HTTP請求的的頻率
frontend ft_web
bind 0.0.0.0:8080
# Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
# Monitors the number of request sent by an IP over a period of 10 seconds
stick-table type ip size 1m expire 10s store gpc0,http_req_rate(10s)
tcp-request connection track-sc1 src
tcp-request connection reject if { src_get_gpc0 gt 0 }
backend bk_web
balance roundrobin
cookie MYSRV insert indirect nocache
# If the source IP sent 10 or more http request over the defined period,
# flag the IP as abuser on the frontend
acl abuse src_http_req_rate(ft_web) ge 10
acl flag_abuser src_inc_gpc0(ft_web)
tcp-request content reject if abuse flag_abuser
server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
server srv2 192.168.1.3:80 check cookie srv2 maxconn 100
4.haproxy的監(jiān)控
hatop是一個(gè)用python語言編寫的,交互式的ncurses客戶端程序。
它的輸出類似top程序,可以用來實(shí)時(shí)查看haproxy的狀態(tài),如果允許level admin則還可以enable,disable服務(wù)器。
[root@haproxy ~]# yum install socat
[root@haproxy ~]# wget http://hatop.googlecode.com/files/hatop-0.7.7.tar.gz
[root@haproxy ~]# tar xvzf hatop-0.7.7.tar.gz
[root@haproxy ~]# cd hatop-0.7.7
[root@haproxy ~]# install -m 755 bin/hatop /usr/local/bin
[root@haproxy ~]# install -m 644 man/hatop.1 /usr/local/share/man/man1
[root@haproxy ~]# gzip /usr/local/share/man/man1/hatop.1
[root@haproxy ~]# vi /etc/haproxy/haproxy.conf
在global段內(nèi)加入如下:
stats socket /var/run/haproxy.stat mode 0600 level admin
重起haproxy
[root@haproxy ~]# /etc/init.d/haproxy reload
確認(rèn)socket已建立
[root@haproxy ~]# ls -al /var/run/haproxy.stat
srw-------. 1 root root 0 Jan 15 20:53 haproxy.sock
運(yùn)行hatop查看haproxy相關(guān)實(shí)時(shí)信息
[root@haproxy ~]# hatop -s /var/run/haproxy.stat
5.用Zabbix監(jiān)控haproxy[http://www.juhonkoti.net/2010/10/15/script-and-template-to-export-data-from-haproxy-to-zabbix]
6.單網(wǎng)卡多個(gè)不同網(wǎng)段的相關(guān)配置
[root@localhost examples]# vi /etc/iproute2/rt_tables
文件結(jié)尾追加如下內(nèi)容:
64 CNC64
202 CNC202
211 CNC211
配置多路由表
[root@haproxy ~]# vi /etc/haproxy/haproxy.conf
#!/bin/bash
######
CNC64_IP="64.4.2.0/24"
CNC64_GW="64.4.2.1"
CNC202_IP="202.108.35.0/24"
CNC202_GW="202.108.1"
CNC211_IP="211.113.58.0/24"
CNC211_GW="211.113.58.1"
ip route flush table CNC64
ip route add default via $CNC64_GW dev eth0 table CNC64
ip rule add from $CNC64_IP table CNC64
ip route flush table CNC202
ip route add default via $CNC202_GW dev eth0 table CNC202
ip rule add from $CNC202_IP table CNC202
ip route flush table CNC211
ip route add default via $CNC211_GW dev eth0 table CNC211
ip rule add from $CNC211_IP table CNC211
修改keepalived配置文件
[root@haproxy ~]# vi /etc/haproxy/haproxy.conf
virtual_ipaddress_excluded { # 超過20個(gè)VIP可以添加在virtual_ipaddress_excluded中,這些VIP不需要發(fā)送檢測包
64.4.2.111/24 dev eth0
202.108.35.22/24 dev eth0
211.113.58.7/24 dev eth0
}
七、SSL offload配置(使用self-signed證書)
]# mkdir /etc/ssl
]# cd /etc/ssl
]# openssl genrsa -des3 -out server.key 1024
]# cp server.key server.key.orig
]# openssl rsa -in server.key.orig -out server.key #去掉pravite key的passphrase
]# openssl req -new -key server.key -out server.csr
>Enter pass phrase for server.key:
>You are about to be asked to enter information that will be incorporated
>into your certificate request.
>What you are about to enter is what is called a Distinguished Name or a DN.
>There are quite a few fields but you can leave some blank
>For some fields there will be a default value,
>If you enter '.', the field will be left blank.
>-----
>Country Name (2 letter code) [XX]:US
>State or Province Name (full name) []:CA
>Locality Name (eg, city) [Default City]:Irvine
>Organization Name (eg, company) [Default Company Ltd]: Monster Inc.
>Organizational Unit Name (eg, section) []:
>Common Name (eg, your name or your server's hostname) []:*.monster.com
>Email Address []:
>
>Please enter the following 'extra' attributes
>to be sent with your certificate request
>A challenge password []:
>An optional company name []:
]# openssl x509 -req -days 365 -in server.csr \
-signkey server.key \
-out server.crt
]# cat server.crt server.key|tee server.pem
haproxy的相關(guān)配置:
frontend localhost
bind *:80
bind *:443 ssl crt /etc/ssl/server.pem
redirect scheme https if !{ ssl_fc }
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
option forwardfor
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server web01 172.17.0.3:9000 check
server web02 172.17.0.3:9001 check
server web03 172.17.0.3:9002 check
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
2017.02.16 補(bǔ)充一個(gè)方便的技巧
haproxy官方提供了針對vim的語法文件,可以高亮顯示keyword,對于修改配置文件來說很方便。
方法說一下:
1.將haproxy源碼中example目錄中的haproxy.vim復(fù)制到$HOME/.vim/syntax/
2.修改$HOME/.vimrc,加入: au BufRead,BufNewFile haproxy* set ft=haproxy
八、系統(tǒng)安全加固
[root@haproxy ~]#yum install yum-remove-with-leaves
[root@haproxy ~]#yum remove gcc make
[root@haproxy ~]#vi remove-list
system-config-firewall-base
iptables-ipv6
dhcp-common
pciutils-libs
efibootmgr
dhclient
kernel-firmware
iwl5150-firmware
iwl6050-firmware
iwl6000g2a-firmware
iwl6000-firmware
ql2400-firmware
ql2100-firmware
libertas-usb8388-firmware
ql2500-firmware
zd1211-firmware
rt61pci-firmware
ql2200-firmware
ipw2100-firmware
ipw2200-firmware
iwl5000-firmware
ivtv-firmware
xorg-x11-drv-ati-firmware
atmel-firmware
iwl4965-firmware
iwl3945-firmware
rt73usb-firmware
ql23xx-firmware
bfa-firmware
iwl100-firmware
b43-openfwwf
aic94xx-firmware
iwl1000-firmware
[root@haproxy ~]#for I in `cat remove-list `;do yum -y remove $i;done
八、參考文檔
1-http://mattiasgeniar.be/2010/11/04/a-custom-init-d-start-up-script-for-haproxy-start-stop-restart-reload-checkconfig/
2-http://www.snapt-ui.com/haproxy/simple-sysctl-tunings-for-haproxy/
3-https://gist.github.com/4039319
4-http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/tproxy.txt
5-http://blog.exceliance.fr/2012/09/10/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
6-http://www.symantec.com/connect/articles/apache-2-ssltls-step-step-part-2
7-http://www.igvita.com/2008/05/13/load-balancing-qos-with-haproxy/
8-http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&tmp_geoLoc=true&docname=c03561757
9-http://www.debuntu.org/how-to-log-haproxy-messages-only-once/#more-713
10-https://transloadit.com/blog/2010/08/haproxy-logging
11-http://kvz.io/blog/2010/08/11/haproxy-logging/
12-https://gist.github.com/1271962
13-http://www.rsyslog.com/doc/rsyslog_conf_actions.html
14-http://tehlose.wordpress.com/2011/10/10/a-log-file-for-each-virtual-host-with-haproxy-and-rsyslog/
15-http://jit.nuance9.com/2009/11/haproxy-routing-by-domain-name.html
16-http://unethicalblogger.com/2010/01/16/virtual-hosting-with-haproxy-and-wsgi.html
17-http://blog.silverbucket.net/post/31927044856/3-ways-to-configure-haproxy-for-websockets
18-http://blog.csdn.net/dog250/article/details/7107537
19-http://www.linuxjournal.com/content/monitoring-processes-kill
20-http://gurucollege.net/technology/ha-lamp-with-keepalived-pt2/
21-http://zauc.wordpress.com/2010/08/31/keepalived-conf之vrrp-instance部分解讀/
22-http://interu.hatenablog.com/entry/20081024/1224784798
23-http://bbs.ywlm.net/thread-845-1-1.html
24-http://heylinux.com/archives/1942.html
25-http://www.intel.com/content/www/us/en/ethernet-controllers/82575-82576-82598-82599-ethernet-controllers-latency-appl-note.html
26-http://blog.csdn.net/turkeyzhou/article/details/7528182
27-http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf
28-http://www.intel.com/support/cn/network/sb/cs-025829.htm
29-http://kaivanov.blogspot.kr/2015/02/keepalived-using-unicast-track-and.html
30-http://www.golinuxhub.com/2013/03/setting-up-custom-tcpip-keep-alive.html
31-https://serversforhackers.com/using-ssl-certificates-with-haproxy
32-https://www.digitalocean.com/community/tutorials/how-to-create-a-ssl-certificate-on-nginx-for-centos-6
33-http://man.lupaworld.com/content/manage/vi/doc/syntax.html