利用MBLB解决TCP长连接负载均衡测试方案.docx

F5 BIGIP MBLB测试记录F5北京杨明非2009年8月目录1. 测试环境31.1测试环境准备31.2测试网络拓扑31.3 BIGIP MBLB 工作原理： 42. V10 MBLB测试过程52.1 TCP连接测试52.2交易分发测试62.3启动第二个客户端的连接建立过程及Timeout82.4加入新的客户端观察负载均衡算法102.5手工Disable服务器测试122.6关闭服务器测试132.7 V10 MBLB 测试总结142.8附：TCPdump数据包分析143. One Connect 工作模式测试163.1 One Connect 模式的工作原理173.2 TCP连接测试173.3交易分发测试193.4 启动第二个客户端的连接203.5启动多个客户端观察负载均衡算法223.6手工Disable服务器测试253.7重新Enable服务器263.8 关闭服务器测试293.9 One Connect 模式测试总结： 304. 附录304.1如何使用iRules来判断交易边界304.2 关于交易定向发送324.3 关于会话保持324.4两种模式的对比334.5还需要研究的部分341. 测试环境1.1测试环境准备PC server 一台，安装 Windows 2003 Server.BIGIP 1台，安装10.0.1版本TCP Client/Server 软件1.2测试网络拓扑所有的IP地址均在同一个网段内，TCP client和Server也运行在同一台设备上。通过 启动多个不同的实例来模拟多台Server和Client。测试用BIGIP配置virtual test_vs snat automappool test_pooldestination 60.247.114.43:9000ip protocol tcp rules mblb-basic profiles mblb tcp tcp_half_openpool test_pool monitor all members 60.247.114.34:9000 60.247.114.34:9001 注意mblb的Profile是手工加入的，在图形界面里没有配置。另外对于这种类型的Server, 最好使用tcp_half_open健康检查模式。rule mblb-basic when CLIENT_ACCEPTED TCP：:collectwhen CLIENT_DATA TCP:releaseTCP:notify request#log "client_data trigered"TCP:collectwhen SERVER_CONNECTED TCP:collectwhen SERVER_DATA TCP:releaseTCP:notify response#log "Server_data trigered"TCP:collect1.3 BIGIP MBLB 工作原理:60-247.114,3460.247.114-3450,247,114.34:900060.247,114,34:9001客户端首先与BIGIP建立TCP连接，在客户端发送数据的时候，BIGIP根据交易将客 户端请求发送到不同的服务器，在发送前，BIGIP将与后台服务器建立连接。在这种工作模 式下，可以支持同步阻塞模式交易或者同连接里的异步交易。同步工作模式：Clientl RequestServerl ResponseClient2 RequestServer2 ResponseClientl RequestServer2 Response异步工作模式：Clientl RequestClient2 RequestClientl RequestServerl ResponseServer2 Response-Server3 Response在异步工作模式下，不能用下面测试的简单irules，需要使用iRules来判断每个交易的 边界，以便将每笔交易请求分发到不同的服务器上。下面的测试基于小包状态，也就是每笔交易的长度不超过1个MTU，通常情况下是1460 字节的情况，在这种情况下，在一次CLIENT_DATA事件触发的时候就可以接收到整个的 交易请求或者交易回应。2. V10 MBLB测试过程2.1 TCP连接测试首先启动两台Server，分别侦听9000和9001端口确认在BIGIP里显示两台服务器都是工作的。Cirrent Mernbern匠 | yj Slslus Membert Node NameRalioPriorfTp Groupo1加洞Q60 247 114.4-900011 (Adr-e：!B conn显示没有任何的链接产生rootltm3600：Active config # b conn60.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/0上面的那个连接是我的SSH登录产生的。启动客户端，配置好发送的内容，点击Connect观察BIGIP上的连接状态：rootltm3600：Active config # b conn60.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:4933 <-> 60.247.114.43:9000 <-> any6 tcp 1/1在客户端没有发送数据之前，在BIGIP上只有一个Client-Any6的连接，此时客户端还 没有发送数据，因此BIGIP与后台并不建立连接。2.2交易分发测试点击客户端上的发送按钮观察客户端的收发状态观察Server端收发状态观察BIGIP上的连接状态rootltm3600：Active config # b connany6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000tcp 1/1any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001tcp 1/160.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/0 60.247.114.34:4933 <-> 60.247.114.43:9000 <-> any6 tcp 1/1可以看到，在客户端开始发送数据后，BIGIP分别和两台Server建立了连接，并将客 户端的请求以轮询的方式发送到两台服务器上。由于我在这里启用了SNAT，因此可以看到第一个Server连接是使用的客户端源端口和服务器建立连接，第二个客户端连接使用的另 外一个源端口和服务器建立连接。2.3启动第二个客户端的连接建立过程及Timeout启动第二个客户端建立连接观察BIGIP状态rootltm3600：Active config # b conn60.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6 tcp 1/0怎么没有Server端连接了呢？看看Server端日志，原来由于俺写文章的时间太长，被BIGIP timeout 了。好，现在就让C2开始发送数据看到Server端又开始建立连接了身cgnticin:T remd datLjfpm the transport connection: An existing connection was forcibly closed by the remote host. Connect: 60.247.114.44:31243>Recive: 60.247.114.44:31243: C260.247.114.34:10885 end: S 2 response60.247.114.44:31243S er?erT ailRecive: S2 respors&6a.247.114.44:3124JS&rverT3ilC260.247.114.34:10886 end: S 2 responseG0.247.114.44:312436 erverT ailRecive: S2 resporse6a.24-7.1 14.44:31243ServerTciilC260.247.114.34:1088S end: S 2 response60.247.114.44:31243S erverT ailE.xr：tri：irrT I IrnhlE fn r仅日ri riut目坷m the transport connection: An enisling connection was forcibly closed bp the remote host.I： Connect: £0.247.114.44:1088J)lTecive: bLl.24/.114.44:lUUy： u560.247.114.34:1088Send: SI Response60.247.11 4.44:10SBRecive; 51 Re?ponse60.247.114.44;1088C260.247.114.34:1 OSSSend: SI Response60.247.114.44:1088Recive: S1 Response60.247.114.44:1088C260.247.114.34:1088Send: SI ResponseG0.247.11 4.44:1098BIGIP上的连接状态:rootltm3600:Active config # b conntcp 1/0tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:900060.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6 tcp 1/0现在开始启动C1发送数据C1的连接也被断掉了：重新启动C1并连接BIGIP上状况：rootltm3600：Active config # b connany6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001tcp1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000tcp1/060.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6tcp1/060.247.114.34:1144 <-> 60.247.114.43:9000 <-> any6tcp1/0当C1开始发送数据的时候：rootltm3600:Active config # b connany6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000 tcp 1/060.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6 tcp 1/060.247.114.34:1144 <-> 60.247.114.43:9000 <-> any6 tcp 1/0Server上的状态：Recive: S2 resporse60.247.114.44:31243ServerTailC260.247.114.34:1088Send: S2 respon£e60.247.1U.44:31243SererTailRecive: S2 resporse60.247.114.44:31245ServerT3ilC1 60.247.114.34:1144Send: S2 re$ponje60.247.114.44:31245Ser/erTailRecife: SI Response60.247.114.44:1088C260.247.114.34:1088Send: SI Re$ponse60.247.114.44:1088Recive: 51 Response60?47.114.44:1144C1 60.247.114.34:1144可以看到BIGIP针对每一个客户端连接，分别在每台Server上建立了同样数量的连接，并将请求在这些连接里进行分发。2.4加入新的客户端观察负载均衡算法我们再启动C3,看看有什么状况？Server IP |GD.2d7.11 d -13Server Fori §5oo|1O3O ms5Ennd(&fi) | Log(81)Data HeadData End tcp Testing ClientBIIGProotltm3600：Active config # b connany6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000tcp1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001tcp1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000tcp1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001tcp1/060.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/0 60.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6tcp1/060.247.114.34:1144 <-> 60.247.114.43:9000 <-> any6tcp1/060.247.114.34:1231 <-> 60.247.114.43:9000 <-> any6tcp1/1当C3开始发送数据的时候：Server IP |dO.24<1 14.43Server Rzirt 30001100J nis5emd(8) | Log(8d_)Data HeadClear LogCbseC|Data End朝 TCP Testing ClientServer 状态：Connect: 60.247.114.44:1231Recivi: 50.247.114.44:1231: C360.247.114.34:1231 Send S1 Response60.247.114.44:1231ERecive: 51 ResponseeO.247.114.44:1231 C360.247.11 4.34:1231 Send: S1 Response60247.114.44:1231Recive: S1 Response60.247.114.44:1231 C360.247.11 4.34:1231 n . j. r 1 . -rrn 7t r t m h.-i nn-i i Connect: 60.247.114.44:31840"Recive: 60.247.114.44:31848: C360.247.114.34:1231 Send: S2 response60.247.114.44:31848ServerTilRecive: S2 response60.247.114.44:318485erverTailC360.247.114.34:1231Send S2 respanse60.247.114.44;31S48SerYerTilRecive: S2 responSe60.247.l14.44:31848SeiverTailC3G0.247.114.34:1 231 r . j. r nrrn n jt 1 1 a n-i mr . .t .;i两台Server都收到了 C3的请求BIGIP 上显示 3 个 client connection, 6 个 Server connection:rootltm3600：Active config # b connany6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000 tcp 1/1any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001 tcp 1/1any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9000 tcp 1/0any6 <-> 60.247.114.43:9000 <-> 60.247.114.34:9001 tcp 1/060.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1088 <-> 60.247.114.43:9000 <-> any6 tcp 1/060.247.114.34:1144 <-> 60.247.114.43:9000 <-> any6 tcp 1/060.247.114.34:1231 <-> 60.247.114.43:9000 <-> any6 tcp 1/1在S2上收到的是C1和C2的请求Recive: S2 re3ponse60247.114.44:31245ServerTailC1 60.247.114.34:1144Send; S2 resPQnse60.247.114.44:31245ServerTailiRecive: S2 response60.247.114.44:31850ServerTailC260.247.114.34:1271Send: S2 reeponeeS0.247.114.44:31850ServerTailRecive: 52 re3ponse60247.114.44:31850ServerTailC260.247.114.34:1271 Send; S2 response60.247.114.44;31850ServerTailRecife: S2 re5pon5&60.247.114.44:31245Serverl3ilC1 60.247.114.34:1144Send: S2 responge60.247.114.44:31245Serv8rT8ilReciYe: S2 respons&60247.114.44:312455erYerTailC1 60.247.114.34:1144Send: S2 reSponsee0.247.114.44:31 245ServerTailRecife: S2 response=G0.247.114.44:31245ServerTsilC1G0.247.114.34:1144在S1上收到的是C1和C3的请求R&cive: S1 Re5pon5e60.247.114.44:1144C1 60.247.114.34:1144Send: S1 Response60.247.114.44:1144IRecive: S1 Respond60.247.11 4.44:1 221 C360.24-7.114.34:1231 Send: S1 Response60.247.114.44:1231 R&cive: S1 Response60.247.114.44:1231 C360.247.114.34:1231 Send S1 ResponseE0.247.114.44:1231Recive: S1 Response60.247.11 4.44:1144C1 60.247.h 14.34:1144Send S1 RespanSe60.247.114.44:1144R&cive: S1 REsponse60.247.11 4.44:1 231 C360.247.114.34:1231 停止所有的客户端，然后全部重新发送的时候,Server端接收发生了变化:S1上收到的是C1和C2的请求Send: 51 Response6a.247.114.44:1271Recive; S1 RespanSe60.247.114.44:1271 C260.247.114.34:127115end: S1 Resp&nseGa.247.114.44:1271Fiecive: S1 Response60.247.114.44:1144C1 60.247.114.34:1144Send S1 Respanse6a.247.114.44:1144Recive: S1 ResponsefGO.247.114.44:1271 C2GO.247.114.34:1 271Send: S1 Respanseea.247.114.44:1271Fiecive: S1 Response60.247.114.44:1271 C260.247.114.34:1271Send S1 Respm?e60.247.114.44:1271Recive: S1 Response60.247.114.44:1144C1 GO.247.114.34:1144S2上收到的是C1和C3的请求7 5 IU . £_ I 口 I. tT I . I I T. TT. I 口JUI "T JI LJIIRecivB：S2response60.24 7.114.44:31 S52ServerTailC360.247.114.34:1231!Send: S2 reSponse60.247.114.44:31 852ServerTailRecive: S2 response6a.247.114.44:31245Serverl3ilC1 EO.247.114.34:1144Send: S2 respons&60.247.114.44:31245SBrvBrTailRecive:S2response60.24 7.114.44:31 S52ServerTailC360.247.114.34:1231Send: S2 responseG0.247.114.44:21 0E2ServerTailReciv&:S2response6a.24 7.114.44:31852Serverl3ilC3E0.247.114.34:1231Recive: S2 response6a.247.114.44:31245ServerTailC1 60.247.114.341144应该是Round Robin的算法导致了这种现象的出现BIGIP上的连接没有发生变化：rootltm3600：Active config # b connany6<->60.247.114.43:9000<-> 60.247.114.34:9001tcp 1/1any6<->60.247.114.43:9000<-> 60.247.114.34:9000tcp 1/0any6<->60.247.114.43:9000<-> 60.247.114.34:9001tcp 1/1any6<->60.247.114.43:9000<-> 60.247.114.34:9000tcp 1/1any6<->60.247.114.43:9000<-> 60.247.114.34:9001tcp 1/0any6<->60.247.114.43:9000<-> 60.247.114.34:9000tcp 1/160.247.98.162:14774 <-> 60.247.114.44:ssh-> 60.247.114.44:ssh tcp 1/060.247.114.34:1144 <->60.247.114.43:9000<->any6tcp1/060.247.114.34:1231 <->60.247.114.43:9000<->any6tcp1/160.247.114.34:1271 <->60.247.114.43:9000<->any6tcp1/12.5手工Disable服务器测试现在手工Disable 一台服务器在S1上收到了 3个客户端的请求Recive: S1 Response60.247.11 444:1144C1 60.247.114.34:1144Send 51 Fl弦pm泪阑,247,114.44; 1144Recive: S1 Response60.247.11 4.44:1 231 C360.247.114.34:1231 Send: 51 RBsponse60.247.114.44:1231 Recive: S1 Response60.247.114.44:1271 C260.247.114.34:1271 Send: S1 ReSponseE0.247.114.44:1271Recive: S1 Response60.247.11444:1144C1 60.247.114.34:1144Send: S1 R&sponse60.247.114.44:1144'Recive: 51 Response60.247.11 4.44:1 231 C360.247.114.34:1231 Send: S1 Response60.247.114.44:1231Recive: S1 Response60.247.114.44:1271 C260.247.114.34:1271 Send S1 Response60.247.114.44:1271Recive: S1 Response60.247.11444:1144C1 60.247.114.34:1144Send: S1 Response60.247.114.44:1144Recive: S1 RespOnSe60.247.11 4.44:1 221 C360.247.114.34:1231 Send: S1 R&sponseE0.247.114.44:1231 Recive: S1 Response60.247.11444:1271 C260.247.114.34:1271 Send S1 ReSpanSe60.247.114.44;12?1R&cive: 51 ResponMGO. 247.11 4.44:1144C1 GO.247.114.34:1144Send: 51 Response60.247.114.44:1144Recive; S1 Respanse60.247.114.44;1144C1 60.247.114.34; 1144Send: S1 R&5ponseE0.247.114.44:1144恢复disable的服务器:S1收到了 C1和C2的请求：Send: SI Responeegb.247.1U.44:1271 ' 'Recive: S1 Response60247.114.44:1144C1 60.247.114.34:1144IS end: SI ResponseG0.247.114.44:1144Recive: S1 R espons&60.247.114.44:1271 C260.247.114.34:1271 Send: SI Response60.247.114.44:1271Recive: S1 R esponseG0.247.114.44:1271 C2G0.247.11 4.34:1271 Send: SI Response60.247.114.44:1271Recive: S1 Response60247.114.44:1144C1 60.247.114.34:1144Send: SI ResponseG0.247.114.44:1144Recive: 51 R esponse60.247.114.44:1271 C260.247.11 4.34:1271 Send S1 RespQnse&0.247.114.44:1271S2重新开始接受请求，收到C1和C3的请求Recive: S2 responseeO.247.114.44:31245ServerTailC1 60.247.114.34:1144S end: S 2 response60.247.114.44:31245S erverT ailRecive: S2 response60.247.11 4.4-4:31852ServerTailC360.247.114.34:1231 Serd: S2 re5ponse60.247.114.44:31852ServerTailRecive: S2 response60.247.11444:318525erverTailC360247.114.34:1231 S end; S 2 response60.247.114.44; 31 S52S erverT ailRecive: S2 response60.247.11 4.44:31245ServerTailC1 60.247.114.34:1144S end: S 2 response60.247.114.44:31245S erverT ailRecive: 52 response60.247.114.44:31852ServerTailC360247.114.34:1231 Send: S2 response60.247.114.44:318525erverTailRecive: S2 response60.247.11444:31 S52Ser7erTailC360.247.114.34:1231 2.6关闭服务器测试关闭S2所有的Client和Server都崩溃了！ ！ ！ ！ ！ ！ ！等待服务器程序的改进版本中。2.7 V10 MBLB测试总结BIGIP V10已经具备了 MBLB的处理能力，可以对长连接里面的TCP交易进行拆分处 理，将不同的请求发送到不同的服务器上，并将服务器的返回信息发送到正确的客户端。目前发现的一些可能存在的问题：1、对于每个客户端的长连接，BIGIP将在每个Server上建立一个连接，也就是说对 于每台Server而言，都会有所有的客户端连接数的总和数量的连接，在实际应 用中，需要确定服务器是否能处理全部客户端连接数量的连接数。2、关于交易的边界定义，目前的测试中非常简单的使用了 CLIENT_DATA和 SERVER_DATA事件，这两个事件默认情况下是每接收一个数据包就触发一次， 因此在交易小于1个MTU，通常情况是1460byte的情况下，可以不用区分交易 边界，默认认为一个数据包就是一次交易。3、如果每次发送交易的长度大于1460，就需要用irules去获取和判断交易的长度。 具体的做法是在第一个数据包进来的时候查询数据包中对于交易长度的定义， 然后判断当前收集到得数据是否是完整的交易，如果完整，则释放请求，如果 不完整，则继续进行收集，直到收集到足够的数据后，释放交易长度的内容到 服务器。4、目前测试的应用时阻塞类型的应用，也就是Client必须等待Server应答之后才开 始发送下一个请求，而且数据包都比较小，肯定在一个packet就发送完毕，因 此不存在有边界界定的问题5、如果有非阻塞型应用，也就是客户端可能一次发出多个请求，在不等待server 回应的情况下可以持续发出请求，Server回应也是不等待的