Load Balancing With Nftables - NetDev Conf

7m ago
1.86 MB
6 Pages
Last View : 2m ago
Last Download : 2m ago
Upload by : Lucca Devoe

Load Balancing with nftablesLaura GarcíaZen Load BalancerSeville, Spainlauragl@sofintel.netAbstractThe motivation to design a load balancer prototype with nftablesis to provide a flexible network management system withcomplete load balancing capabilities for Linux-based systems, butalso improve Layer 4 load balancing performance using thenftables infrastructure. The iptables approach in this topic lacksfeatures for a complete and high performance load balancingsystem and those shortcomings have been taken into account inorder to be solved in nftables.KeywordsLoad Balancing, nftables, conntrack, netfilter, Linux networking,iptables, LVS.IntroductionLVS allows very easy deployment of Linux-based loadbalancers. Probably less well-known is the fact that youcan also use iptables rules using the existing matches andtargets to implement many of the core load balancingfeatures such as different scheduling approaches anddispatching methods, flow persistence, etc.This paper discusses the implementation of a Linuxbased load balancers using iptables, we will describe ourruleset configurations, lessons learned from integrationissues with Netfilter and other networking software andexisting limitations. Moreover, we have planed a prototypebased on nftables, detailing what is missing and what weconsider good to have to improve its load balancingcapabilities.Load Balancing SolutionsThe most popular and extended solution currently availablefor load balancing at layer 4 is Linux Virtual Server. Lesswell known solution but provides a very good results is toperform load balancing using iptables extensions. Finally,such iptables approach and knowledge gathered will beused to present a design of a high performance loadbalancing prototype with nftables.LVSLVS is a wide used load balancer at layer 4 which providesa full set of complete and versatile schedulers, severalforwarding methods like Direct Routing, tunneling andsNAT, and additionally some basic integrated healthchecks. LVS provides an additional layer on top of netfilterand it's mostly kernel code base with an user space daemonfor control. In some cases, it's needed to use iptables tomark packets and the support of content parsing isperformed using additional modules than iptables.iptablesLoad balancing with iptables implies the use of the xtablesextensions in order to build a set of rules that behave withthe desired scheduler and forwarding method. Theavailable forwarding methods includes sNAT and dNAT,according to the transparency required in a certaininfrastructure.The mechanism used in this case requires marking thepackets and then forwarding to the determined backend orreal server.The backend health checks needs to be performed fromuser space as a daemon at different layers (icmp, protocolor application checks, etc.), as it's show in the Figure 1.Figure 1. Load balancing with iptables approach.With this approach, all the complexity of rulesetmanagement and health checks are moved to user spaceand in the kernel only remains the packet handling,obtaining very good results regarding the performancerequired. But some concerns could raise into a problemsuch as: The ruleset is handled sequentially which couldproduce not desired side effects during the packethandling. The marking and forwarding rules must besynchronized in order to behave as expected.nftablesFor all these reasons, and following the approachdescribed, we propose to supply the features andProceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

improvements needed within nftables in order to provide afull featured load balancing tool.Load balancing with nftables is possible through thenftables infrastructure: nft libraries, nftables virtualmachine and it's instructions. Inherent properties likedynamic ruleset and atomicity through nft scripting aremajor keys regarding the reliability in this design andavoid side effects concerns that raised the approach withiptables.Another enhancement in terms of performance andthroughput is the fact that matching packets is not neededanymore as it's shown in the Figure 2, thanks to thedynamic ruleset and maps structures.Regarding the forwarding methods available in nftables it'spossible to provide sNAT and dNAT as the iptablesapproach.Forwarding methodsThe required forwarding methods to be implemented atlayer 4 are basically the following in terms oftransparency: sNAT (transparency off) dNAT (transparency on)Health checksThe backends or real servers monitoring will be performedin user space and at different layers, configurable regardingthe application or protocol used.Good integrationThe good integration with other features such as QoS andfiltering could be taken into account for every loadbalancing service.Use CasesSome use cases with all these three solutions presentedwith the basic and required features are shown in thissection, according to the example in the Figure 3.Figure 2. Load balancing with nftables design .In this case, the user space daemon takes care of all thecomplexity of the health checks at different layers and theruleset management, but in this case some concerns aresolved: The ruleset is handled atomically through a nftruleset batch and loaded by the nft virtualmachine. Only it's needed to manage the nat table, avoidingmarking packets.Features to accomplishThe basic features that this new load balancing system withnftables should provide in this first prototype are describedbelow.Figure 3. Use cases environment.Round Robin LB with LVSAccording to the given scenario, a simple load balancingservice can be created with the 3 commands shown below.ipvsadm -A -t -s rripvsadm -a -t -r -mSchedulersThe schedulers most used and required in this prototypeare: Round Robin. Weight. Least Connections.PersistenceThe persistence is required in this kind of technology, atleast: Persistence per source IP address.ipvsadm -a -t -r -mThe first command creates the virtual service over a certainIP address and one port with a round robin scheduler. Thelast 2 commands add the backends into the virtual servicecreated.Round Robin LB with iptablesWith the iptables approach, a simple load balancer can bebuilt with these two rules shown below.Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

iptables -t nat -A PREROUTING -m statistic --mode nth \--every 2 --packet 0 -d -p tcp \setting a certain weight per backend, with values 100 and50 respectively in this case.--dport 80 -j DNAT --to-destination -t nat -A PREROUTING -m statistic --mode nth \--every 2 --packet 1 -d -p tcp \--dport 80 -j DNAT --to-destination ruleset changes the prerouting chain in the nat table inorder to forward the packets detected in the given virtualservice under a certain conditions. In this case, the nthextension is used to match the packets every 2 packets pereach backend.Round Robin LB with nftablesThe nftables proposal is to have, for this simple case, justone rule to build a load balancer.Once the table lb and the chain prerouting are created andassociated to the nat prerouting hook, with just one rule wecan build a round robin scheduler over a certain virtualservice over one IP address and one TCP port, as shown inthe commands below.table ip lb {chain prerouting {type nat hook prerouting priority 0; policy accept;ip daddr tcp dport http dnat nth 2 map {0:,1:}Weight LB with iptablesIn this case, the iptables approach is able to perform aweighted scheduling method using the statistic extension,random mode and with the appropriated probability perbackend, converting a certain weight to probability with aneasy algorithm.iptables -t nat -A PREROUTING -m statistic \--mode random --probability 1 -d \-p tcp --dport 80 -j DNAT \--to-destination -t nat -A PREROUTING -m statistic \--mode random --probability 0.33 -d \-p tcp --dport 80 -j DNAT \--to-destination ruleset changes the prerouting chain in the nat table inorder to forward the packets detected in the given virtualservice. One rule per backend will be enough in this simpleuse case using the statistic extension to match the packetsrandomly with the probability calculated.The first backend will have twice the weight of the secondbackend, for this reason, it's used a probability of 1 tomatch the first rule and the rest 0.33 will pass through thesecond rule.The first rule will ensure that all the packets through thevirtual service will be marked.}}Weight LB with nftablesIn this rule the virtual service is associated to thedestination IP address and TCP port indicated in the givenscenario. Then, it's needed to set the forwardingmechanism to dNAT and the scheduler method to nth every2 packets. The map created will allow to associate thepacket numbering with the backend or real server to beused.Note that by the time this paper is written, the nthinstruction is not consolidated in nftables yet, so the syntaxcould change.In this use case, nftables is able to provide a weightedscheduler setting up one dynamic rule.As the last use case, once the table lb and the chainprerouting are created and associated to the nat preroutinghook, with just one rule we can build a weigthed schedulerover a certain virtual service over one IP address and oneTCP port, as shown in the commands below.table ip lb {chain prerouting {type nat hook prerouting priority 0; policy accept;ip daddr tcp dport http dnat \Weight LB with LVSrandom upto 100 map {The commands below are used in LVS to create a virtualservice with a weighed scheduling method.0-66:,67-99:}ipvsadm -A -t -s wrripvsadm -a -t -r \}}-m -w 100ipvsadm -a -t -r \-m -w 50The first command creates the virtual service over a certainIP address and one port with a weight scheduler. The last 2commands add the backends into the virtual service createdIn this rule the virtual service is associated to thedestination IP address and TCP port indicated in the givenscenario. Then, it's needed to set the forwardingmechanism to dNAT and the scheduler method to random.The map created will allow to associate the random rangeProceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

according to the weight for every backend with the realserver to be used.Note that by the time this paper is written, the randominstruction is not consolidated in nftables yet, so the syntaxcould change.This weighted scheme in nftables will be used as a base inorder to create more complex weighted schedulers, as it'sdescribed in the following sections.Weight LB Multiport with LVSThe multiport case in LVS implies the use of iptables inorder to mark the packets that matches the virtual serviceenabling the multiport extension, as it's shown below.iptables -A PREROUTING -t mangle -d \0-66:,67-99:}}}Note that by the time this paper is written, the randominstruction is not consolidated in nftables yet, so the syntaxcould change.Weight LB IP Persistence with LVSThe persistence is required in many applications and LVSintegrates it quite easily associating the client source IPaddress to a certain backend during a configured timeout,as it's shown below.-p tcp -m multiport --dports 80,443 -j MARK \--set-mark 1ipvsadm -A -t -s wrr -p 300ipvsadm -a -t -r \ipvsadm -A -f 1 -s wrr-m -w 100ipvsadm -a -f 1 -r -m -w 100ipvsadm -a -f 1 -r -m -w 50The iptables command in the mangle table takes care ofmatching and marking the packets that applies to themultiport virtual service and the given IP address.The mark value assigned to the matched packets will beused in order to create the virtual service with the weightedscheduler. After that, it'll be needed to add the backendsinto the new virtual service with the weight value for eachone.Weight LB Multiport with iptablesThe iptables approach with multiport is similar to the lastuse case, as the multiport match is completely compatiblewith the ruleset presented.iptables -t nat -A PREROUTING -m statistic \ipvsadm -a -t \-m -w 50The last parameter indicates that the persistence timeoutwill be 300 seconds.Weight LB IP Persistence with iptablesThe IP persistence approach with iptables is complex as it'snot supported natively. In order to be able to associatesource IP addresses with backends it's used the recentextension, creating one source IP addresses list perbackend.In this complex use case with iptables, two steps will beneeded, one step to mark the new connections packets,through the mangle table, and then a second step toforward the packets according to the mark through the nattable, where every mark will be associated to a certainbackend, as it's shown below.--mode random --probability 1 -d \-p tcp -m multiport --dports 80,443 -j DNAT \--to-destination -t nat -A PREROUTING -m statistic \--mode random --probability 0.33 -d \-p tcp -m multiport --dports 80,443 -j DNAT \--to-destination -t mangle -A PREROUTING -j CONNMARK \--restore-markiptables -t mangle -A PREROUTING -m statistic \--mode random --probability 1 -d \-p tcp --dport 80 -j MARK --set-xmark 1iptables -t mangle -A PREROUTING -m statistic \--mode random --probability 0.33 -d \The multiport match will be required in every backendrule.-p tcp --dport 80 -j MARK --set-xmark 2iptables -t mangle -A PREROUTING -m recent \--name "mark1 list" --rcheck --seconds 120 \Weight LB Multiport with nftablesNftables supports natively multiport capabilities, so it's assimple as including the port list in the weight use case.--set-xmark 1iptables -t mangle -A PREROUTING -m recent \--name "mark2 list" --rcheck --seconds 120 \-d -p tcp --dport 80 -j MARK \table ip lb {--set-xmark 2chain prerouting {type nat hook prerouting priority 0; policy accept;ip daddr tcp dport { http,https } dnat \random upto 100 map {-d -p tcp --dport 80 -j MARK \iptables -t mangle -A PREROUTING -m state --state NEW \-j CONNMARK --save-markiptables -t nat -A PREROUTING -m mark --mark 1 -j DNAT \Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

-p tcp --to-destination -m recent \--name "mark1 list" --setiptables -t nat -A PREROUTING -m mark --mark 2 -j DNAT \-p tcp --to-destination -m recent \--name "mark2 list" --setThe mangle rules ensure that the packets are going to bemarked following the weighted scheduling method withthe weight selected per backend. This is performed withthe statistic match and random mode.After that, it's needed to check if the IP address has alreadybeen used and stuck to any backend. The match recent isused to generate persistence creating one source IP list perbackend with a certain timeout. If during the IP source listchecking the IP is found, then the packet will be markedwith the mark of the selected backend.Finally, the packet will be forwarded through the nat tableto the backend determined in the mark and store in the listthe new IP address entry if needed.For this approach, three rules will be needed per backend:ensure packet mark, check client persistence and forwardpacket storing the IP address in the list.But this iptables approach with IP persistence providesseveral concerns like: The packet must pass through several manglerules until it determines the most affordablebackend, with a complexity of 3n. The need of one list per backend implies to checkevery list and perform several lookups which isexpensive.Weight LB IP Persistence with nftablesThe persistence in nftables is not natively integrated but itcould be easily built using dynamic maps. With nftablessome concerns regarding the iptables approach are going tobe solved. Firstly, the packets are not needed to be markedin order to be forwarded and there is only one list pervirtual service instead of per backend. With this prototype,it's easy to configure but also we could get much moreperformance.Once the new lb table is created, it's needed to add adynamic map where the association between clients sourceIP addresses and backend are going to be stored with atimeout, in this case it's called dnat-cache.The chain cache-done is executed once the IP matches inthe list in order to perform directly the forward to thebackend associated. The lookup it's going to be performedquite fast in this case.The prerouting chain provides all the scheduling logic andthe source IP cache list maintenance. Firstly, it'll be neededto link the chain with the corresponding hook. Then, checkin the cache list if the IP already exists in the list and jumpto the cache-done chain without return.After that, it's needed to apply the dynamic rule whichdetermines the backend to be selected for any newconnection following the weighted base, as it has beenshown in the cases before.Finally, add in the dynamic map the IP address the newassociations between source IP address and backend.Note that by the time this paper is written, the instructionsare not consolidated in nftables yet, so the syntax couldchange.This weighted scheme with IP persistence in nftables willbe used as a base in order to create more complex weightedschedulers with IP persistence, as it's described in thefollowing sectionsWeighted Least Connections LB with nftablesThe weighted nftables schema shown before can be used asa base to build more complex scheduling methods likeweighted least connections. In this use case, the user spacedaemon is able to gather the number of connections forevery backend from the conntrack as it's show in theFigure 4. The user space daemon updates the weight forevery backend according to this number of establishedconnections, more established connections to a certainbackend implies less dynamic weight assigned.table ip lb {map dnat-cache { type ipv4 addr : ipv4 addr; \timeout 120s; }chain cache-done { dnat ip saddr map @dnat-cache }chain prerouting {type nat hook prerouting priority 0; policy accept;ip saddr @dnat-cache goto cache-doneip daddr tcp dport http dnat \random upto 100 map {Figure 4. Weighted Least Connections LB with nftables .0-66:,Weighted Least Response LB with nftables67-99:}map dnat-cache add { ip saddr : ip daddr }}}In this use case, the user space daemon is able to gather theresponse time for every health check performed againstevery backend. All this valuable information is used toestimate the best weight for every backend, more responsetime spend for a backend implies less dynamic weightassigned.Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

The Figure 5 represents the behavior of this use case. Simplify the kernel infrastructure, as according tothis prototype the complexity is moved to userspace.Consolidate the kernel development, as nftablescould join the efforts in order to avoid duplicatedwork, better maintenance and native LB support.Unique API for network management, as nftablescould be able to provide an user interface forfirewalling, QoS and load balancing as well.Even more than that, it's able to build a high performanceload balancer with nftables.Figure 5. Weighted Least Response LB with nftables .Weighted Least CPU Load LB with nftablesThe weighted Least CPU Load use case needs to gather theCPU Load of every backend in order to estimate thedynamic weight for each one of them.In this use case, the user space daemon is able to gather theCPU load through SNMP checks against the backends,estimating the dynamic weight where more load impliesless weight.The Figure 6 represents the behavior of this use case.AcknowledgementsFrom the Zen Load Balancer Team, we would like to thankPablo Neira for his support during the preparation of thistalk and mentoring to implement this prototype.Bibliography1. Nftables wiki, http://http://wiki.nftables.org2. Zen Load Balancer entation/Author BiographyLaura García studied Computer Science in the Universityof Seville and she has been a Software Engineer for HPand Schneider Electric. Over 10 years of experience withembedded Linux systems. Currently, she is CEO and cofounder of Sofintel IT Engineering SL company in order tocontinue the development and evolution of the open sourceproject Zen Load Balancer.Figure 6. Weighted Least CPU load LB with nftables .Similar cases could be performed through SNMP schedulerfor memory, network consumption, etc.Work to doThis prototype is still under design and some developmentswill be needed in order to be able to fulfill therequirements to provide load balancing with nftables: Implement some native functions in nftables:random and nth instructions and some mapsenhancements. User space daemon nft-lbd: health checks support,dynamic weight (least connections, least response,etc.)ConclusionsThe final conclusions regarding the prototype of loadbalancing with nftables are:Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain

for load balancing at layer 4 is Linux Virtual Server. Less well known solution but provides a very good results is to perform load balancing using iptables extensions. Finally, such iptables approach and knowledge gathered will be used to present a design of a high performance load balancing prototype with nftables. LVS LVS is a wide used load .

Related Documents:

8. Load Balancing Lync Note: It's highly recommended that you have a working Lync environment first before implementing the load balancer. Load Balancing Methods Supported Microsoft Lync supports two types of load balancing solutions: Domain Name System (DNS) load balancing and Hardware Load Balancing (HLB). DNS Load Balancing

Internal Load Balancing IP:, Port: 80 Web Tier Internal Tier Internal Load Balancing IP:, Port: 80 asia-east-1a User in Singapore Database Tier Database Tier Database Tier External Load Balancing Global: HTTP(S) LB, SSL Proxy Regional: Network TCP/UDP LB Internal Load Balancing ILB Use Case 2: Multi-tier apps

It is used for Balancing the load according to controller and according to flow of Data as well. Data Plane handle Link Load Balancing and Server Load Balancing. The Distributed multiple control architecture is subcategorized into Flat Architecture and hierarchical Architecture. It helps to explore new dimensions of load balancing. Figure 4.

load balancing degree and the total time till a balanced state is reached. Existing load balancing methods usually ignore the VM migration time overhead. In contrast to sequential migration-based load balancing, this paper proposes using a network-topology aware parallel migration to speed up the load balancing process in a data center.

load balancing. The load balancing framework in CHARM is based on a heuristic known as the principle of persistence [8] which states that the recent past is a good indication of the future. CHARM provides the application programmer with a suite of load balancers and the capability to add new custom load balancing strategies. These load .

Load Balancing Solutions - iptables ruleset mng & health daemon BACKEND 0 BACKEND 1 prerouting mangle prerouting nat check_ping, check_tcp, check_http, . iptables load balancer user space kernel space pkt (1st Approach) ! " # " % & ' & ( )

In general, a dynamic load-balancing algorithm consists of four major components: the load measurement rule, the information exchange rule, the initiation rule, and the load balancing operation [8], [10]. 3.1 Load measurement . In order to quantify the load information of a node, we take a variable called load index.

Introduction to Qualitative Field Research 3 01-Bailey-(V-5).qxd 8/14/2006 6:24 PM Page 3. He observed, interviewed, and took photographs of them, even one of “Primo feeding cocaine to Caesar on the benches of a housing project courtyard” (p. 101). Purpose of Research and Research Questions Although all field research takes place within natural settings, it serves different purposes. It is .