Keepalived+Lvs 原来如此简单，so easy

Keepalived+Lvs 原来如此简单，so easy

前言

在lvs架构中，当后端的rs宕掉时，调度器仍会把请求转发到宕掉的rs上，而使用keepalived就可以解决该问题。

keepalived已经嵌入了lvs功能，无需使用ipvsadm & 无需编写lvs相关脚本。完整的keepalived+lvs需要两台调度器实现高可用，提供调度服务的只需要一台，另外一台作为备用。

用了lvs加keepalived可以使用一个VIP，LVS和keepalived整合的话，哪我们不需要手工去配置VIP操作了

[root@localhost ~]# ipvsadm -A -t 192.168.179.199:80 -s rr[root@localhost ~]# ipvsadm -a -t 192.168.179.199:80 -r 192.168.179.100 -g -w 100

这些步骤都不再需要了，只需要ipvs模块就行了，即有ipvsadm管理命令就行了

结构： 主keepalived（调度器）：192.168.179.102 备keepalived（调度器）：192.168.179.103

真实服务器rs1：192.168.179.99 真实服务器rs1：192.168.179.104

VIP：192.168.179.199 lvs为dr模式，rs主机上运行着提供80服务的nginx

keepalived 配置文件结构

vrrp_instance VI_1

vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 100 advert_int 5 nopreempt authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.179.199 }}# Virtualserver也用这个VIPvirtual_server 192.168.179.199 80 {}

virtual_server

virtual_server 192.168.1.199 80 {这个是LVS的配置 后面指定的是VIPlb_algo wrr 算法lb_kind DR 指定转发方式是DR模式virtual_server 192.168.179.199 80 { delay_loop 6 #监控检查时间间隔，keepalived可以检查后端机器的服务状态(Keepalived会每隔6秒去探测后端real srver的80端口是不是能够通，不通踢出LVS均衡，如果正常访问会加到均衡里面来) lb_algo rr #调度算法，一般是rr模式 lb_kind DR #转发方式 persistence_timeout 60 #会话保持时间，五秒之内用户请求都在后端一台机器上 protocol TCP

后面一个real server配置一个段，要将后端的真实IP写进去，有多少个机器加多少个

real_server real_server 192.168.33.12 80 { weight 100 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 }}

Ipvsadmin还需要安装，安装目的是通过-Ln查看

配置真实服务器real-server

在192.168.179.99，192.168.179.104 nginx服务器上怕配置VIP和抑制ARP请求

简单说一下:  假设A为前端负载均衡服务器， B,C为后端真实服务器。 A接收到数据包以后，会把数据包的MAC地址改成B的（根据调度算法，假设发给B服务器），然后把数据包重新发出去，交换机收到数据包根据MAC地址找到B，把数据包交给B。 这时B会收到数据包，同时验证请求IP地址，由于数据包里的IP地址是给A的，所以正常情况下B会丢弃数据包，为了防止这种情况，需要在B机器的回环网卡上配置A的IP地址。并设置ARP压制。

通过下面脚本在lo网卡上配置VIP和抑制ARP请求

[root@localhost ~]# cat real-server.sh #!/bin/bash #description : start realserverVIP=192.168.179.199case "$1" instart) echo "start LVS of REALServer" /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce;;stop) /sbin/ifconfig lo:0 down echo "close LVS EEALServer" echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce;;*) echo "Usage: $0 {start|stop}" exit 1esac[root@localhost ~]# ./real-server.sh startstart LVS of REALServer# 输入ifconfig查看虚拟IP是否已经绑定到回环网卡[root@localhost ~]# ifconfig lo:0: flags=73 mtu 65536 inet 192.168.179.199 netmask 255.255.255.255 loop txqueuelen 1 (Local Loopback)# 在lvs机器上192.168.179.102上ping192.168.179.199可以看到ARP被抑制了[root@localhost ~]# ping 192.168.179.199PING 192.168.179.199 (192.168.179.199) 56(84) bytes of data.From 192.168.179.102 icmp_seq=1 Destination Host UnreachableFrom 192.168.179.102 icmp_seq=2 Destination Host UnreachableFrom 192.168.179.102 icmp_seq=3 Destination Host Unreachable^C--- 192.168.179.199 ping statistics ---5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4029mspipe 4

配置Keepalived

Master 192.169.179.102 配置

[root@localhost ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs { router_id LVS_DEVEL}# VIP1vrrp_instance VI_1 { state MASTER interface ens32 virtual_router_id 51 priority 100 advert_int 5 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.179.199 }}virtual_server 192.168.179.199 80 { # 这里配置虚拟服务器 （192.168.179.199 为虚拟IP， 监听80端口）客户端就访问这个地址 delay_loop 6 # 健康检查时间间隔，单位是秒 lb_algo rr # 负载均衡调度算法，互联网应用常使用 wlc 或 rr lb_kind DR # 负载均衡转发规则。一般包括 DR,NAT,TUN3 种，在我的方案中，都使用DR 的方式。 persistence_timeout 60 # 会话保持时间，单位是秒。这个会话是网络7层传输协议中的（第三层）会话。比如我们把它设置为60，那么客户端60秒内访问LVS虚拟服务器的请求都会被转到到同1个后台服务器节点。如果设置为0，那么每次请求都会根据配置的调度情况进行分发。 protocol TCP # 转发协议 protocol，一般有 tcp 和 udp 两种。实话说，我还没尝试过 udp 协议类的转发。 real_server 192.168.179.104 80 { # 这里配置真实服务器的地址与端口 weight 100 # 权重，不解释了。一般都是机器性能好的高一些，低的小一些。 TCP_CHECK { connect_timeout 10 # 连接超时时间 nb_get_retry 3 # 重连次数 delay_before_retry 3 # 重连间隔时间 connect_port 80 # 检测端口 } } real_server 192.168.179.99 80 { weight 100 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 } }}

启动Master的keepalived(最重要的就是查看日志和使用ipvsadm工具来查看)

[root@localhost ~]# tail -f /var/log/messagesNov 25 10:46:57 localhost Keepalived_vrrp[38768]: VRRP_Instance(VI_1) Entering MASTER STATENov 25 10:46:57 localhost Keepalived_vrrp[38768]: VRRP_Instance(VI_1) setting protocol VIPs.Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens32 for 192.168.179.199Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:46:57 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:00 localhost Keepalived_healthcheckers[38767]: TCP connection to [192.168.179.99]:80 success.Nov 25 10:47:00 localhost Keepalived_healthcheckers[38767]: Adding service [192.168.179.99]:80 to VS [192.168.179.199]:80Nov 25 10:47:00 localhost Keepalived_healthcheckers[38767]: Gained quorum 1+0=1 <= 100 for VS [192.168.179.199]:80Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens32 for 192.168.179.199Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:02 localhost Keepalived_vrrp[38768]: Sending gratuitous ARP on ens32 for 192.168.179.199Nov 25 10:47:06 localhost Keepalived_healthcheckers[38767]: TCP connection to [192.168.179.104]:80 success.Nov 25 10:47:06 localhost Keepalived_healthcheckers[38767]: Adding service [192.168.179.104]:80 to VS [192.168.179.199]:80# 可以看到通过健康检查加入了后端[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 0 0 # 单独的keepalived是没有健康检查的，单独的keepalive是用shell脚本来实现健康检查的，这里用到的是keepalived内部的模块，通过TCP端口监测，如果是mysql就监测3306端口，如果是nginx那么就监测80端口# delay_loop 6每隔6s去探测主机80端口是否通 ，如果不通将real server剔除出去，如果又可以正常访问就就加到均衡里面来

[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 2 0

192.168.179.103 slave 配置

[root@localhost ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs { router_id LVS_DEVEL}# VIP1vrrp_instance VI_1 { state BACKUP interface ens32 virtual_router_id 51 priority 50 advert_int 5 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.179.199 }}virtual_server 192.168.179.199 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 60 protocol TCP real_server 192.168.179.104 80 { weight 100 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 192.168.179.99 80 { weight 100 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 } }}[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 0 0

上面整个配置下来可以看到LVS和keepalived整合只是加了一段代码virtual_server {real_server}

健康检查 TCP方式检查

来看看下面的作用

TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 80 }

现在后端两台nginx正常

[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 0 0

现在来到192.168.179.99将nginx干掉

[root@localhost ~]# pkill nginx

再来看看主备keepalived的日志

MasterNov 25 11:17:50 localhost Keepalived_healthcheckers[38767]: TCP connection to [192.168.179.99]:80 failed.Nov 25 11:17:53 localhost Keepalived_healthcheckers[38767]: TCP connection to [192.168.179.99]:80 failed.Nov 25 11:17:53 localhost Keepalived_healthcheckers[38767]: Check on service [192.168.179.99]:80 failed after 1 retry.Nov 25 11:17:53 localhost Keepalived_healthcheckers[38767]: Removing service [192.168.179.99]:80 from VS [192.168.179.199]:80SlaveNov 25 11:17:52 localhost Keepalived_healthcheckers[7827]: TCP connection to [192.168.179.99]:80 failed.Nov 25 11:17:52 localhost Keepalived_healthcheckers[7827]: Check on service [192.168.179.99]:80 failed after 1 retry.Nov 25 11:17:52 localhost Keepalived_healthcheckers[7827]: Removing service [192.168.179.99]:80 from VS [192.168.179.199]:80# 可以看到健康检查失败，会将其踢出后端节点Master[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.104:80 Route 100 0 0 Slave[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.104:80 Route 100 0 0

现在将nginx拉起来

[root@localhost ~]# nginx

再来看看keepalived的日志

MasterNov 25 11:21:54 localhost Keepalived_healthcheckers[38767]: TCP connection to [192.168.179.99]:80 success.Nov 25 11:21:54 localhost Keepalived_healthcheckers[38767]: Adding service [192.168.179.99]:80 to VS [192.168.179.199]:80SlaveNov 25 11:21:55 localhost Keepalived_healthcheckers[7827]: TCP connection to [192.168.179.99]:80 success.Nov 25 11:21:55 localhost Keepalived_healthcheckers[7827]: Adding service [192.168.179.99]:80 to VS [192.168.179.199]:80Master[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 0 0 slave[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 0 0 #健康检查通过，将之前踢出的机器又重新加回来了

健康检查 HTTP_GET方式检查

HTTP_GET：配置健康检查

real_server 192.168.179.104 80 { weight 100 HTTP_GET { url { path /index.html status_code 200 } nb_get_retry 3 delay_before_retry 3 connect_timeout 5 } }connect_timeout 5 ：连接超时时间nb_get_retry 3 ：重连次数delay_before_retry 3 ：重连间隔时间connect_port 8080 ：检测端口

通过上诉命令计算digest  然后写到配置文件上就可以了。

Keepalived Master发生故障（故障转移）

Master(如果将keeplived关闭，那么LVS也不可用了)

[root@localhost ~]# systemctl stop keepalived[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn

Backup(可以看到vip发生了飘移)

[root@localhost ~]# ip a1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: ens32: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:61:90:c1 brd ff:ff:ff:ff:ff:ff inet 192.168.179.103/24 brd 192.168.179.255 scope global ens32 valid_lft forever preferred_lft forever inet 192.168.179.199/32 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::f54d:5639:6237:2d0e/64 scope link valid_lft forever preferred_lft forever#继续对外提供服务[root@localhost ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.179.199:80 rr persistent 60 -> 192.168.179.99:80 Route 100 0 0 -> 192.168.179.104:80 Route 100 2 0

通过上面我相信你对kkepalived+lvs有个清晰了解，本片文章就介绍到这。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。