MikroTik RouterOS Workshop Load Balancing Best Practice

MikroTik RouterOS WorkshopLoad BalancingBest PracticeWarsawMUM Europe 2012 MikroTik 2012

About MeJānis Meģis, MikroTikJānis (Tehnical, Trainer, NOT Sales)Support & Training Engineer for almost 8 yearsSpecialization: QoS, PPP, Firewall, RoutingTeaching MikroTik RouterOS classes since 2005 MikroTik 20122

Load BalancingLoad Balancing is a technique to distribute theworkload across two or more network links inorder to maximize throughput, minimiseresponse time, and avoid overloadUsing multiple network links with load balancing,instead of single network links, may increasereliability through redundancy MikroTik 20123

Types of Load BalancingSub-Packet Load Balancing (MLPPP)Per Packet Load Balancing (Bonding)Per Connection Load Balancing (nth)Per address-pair Load Balancing (ECMP, PCC,Bonding)Custom Load Balancing (Policy Routing)Bandwidth based Load Balancing(MPLS RSVP-TE Tunnels) MikroTik 20124

Multi-Link PPPPPP Multi-link Protocol allows to divide packetequally and send each part into multiplechannelsMLPPP can be created:over single physical link – where multiple channelsrun on the same link (anti-fragmentation)over multiple physical links - where multiplechannels run on the multiple link (load balancing)MLPPP must be supported by both ends(MLPPP is legacy stuff from modem era) MikroTik 20125

MLPPP configurationServer must haveMLPPP supportAll lines musthave same username andpasswordRouterOS hasonly the MLPPPclientimplementation MikroTik 20126

BondingBonding is a technology that allows you toaggregate multiple Ethernet-like interfaces intoa single virtual link, thus getting higher datarates and providing fail-overBonding (load balancing) ce-alb MikroTik 20127

802.3ad802.3ad mode is an IEEE standard also calledLACP (Link Aggregation Control Protocol). MikroTik 20128

Balance-rr and balance-xorBalance-rr mode uses Round Robin algorithm packets are transmitted in sequential order fromthe first available slave to the last.When utilizing multiple sending and multiplereceiving links, packets often are received out oforder (problem for TCP)Balance-xor balances outgoing traffic across theactive ports based on a hash from specificprotocol header fields and accepts incomingtraffic from any active port MikroTik 20129

Balance-tlbThe outgoing traffic isdistributed accordingto the current loadIncoming traffic is notbalancedThis mode is addresspair load balancingNo additionalconfiguration isrequired for the switch MikroTik 201210

Balance-albIn short alb tlb receive loadbalancingThis mode requires adevice drivercapability to changethe MAC address MikroTik 201211

ECMP RoutesECMP (Equal CostMulti Path) routeshave more than onegateway to the sameremote networkGateways will beused in Round Robinper SRC/DSTaddress combinationSame gateway can bewritten several times!! MikroTik 201212

“Check-gateway” OptionYou can set the router to check gatewayreachability using ICMP (ping) or ARP protocolsIf the gateway is unreachable in a simple route– the route will become inactiveIf one gateway is unreachable in an ECMProute, only the reachable gateways will be usedin the Round Robin algorithmIf Check-gateway option is enabled on oneroute it will affect all routes with that gateway. MikroTik 201213

Interface ECMP RoutingIn case you have more that one PPP connectionfrom the same server, but MLPPP is impossible(different user names, server support missing) itis possible to use Interface routingSimple IP address routing is impossible for allPPP connections that have the same gatewayIP addressTo enable interface routing just specify all PPPinterfaces as route gateway-interfacesWorks only on PPP interfaces. MikroTik 201214

ECMP and MasqueradeAs forwarding database is rebuilt every 10min inLinux Kernel, there is a chance that connectionwill jump to the other gatewayIn the case of masquerading this jump results ina change of source address and in eventualdisconnectMore info cache-dos.htmlhttp://marc.info/?m inux/net/0305.2/index.html#19 MikroTik 201215

Configuration Setup MikroTik 201216

Basic Configuration MikroTik 201217

Policy RoutingPolicy routing is a method that allows you tocreate separate routing polices for differenttraffic by creating custom routing tablesIn RouterOS these routing tables are created:For every table specified in /ip route ruleFor every routing-mark in mangle facilityMarked traffic is automatically assigned to theproper routing table (no need for lookup rules) MikroTik 201218

Routing-markRouterOS attribute assigned to each packetRouting-mark can be changed in firewall manglefacility just before any routing decision:chain Prerouting – for all incoming trafficchain Output – for outgoing traffic from routerEvery new routing mark has its own routingtable with the same nameBy default all packets have the “main” routingmark MikroTik 201219

Traffic to Connected NetworksAs connected routes are available only in “main”routing table, it is necessary that traffic toconnected networks stay in “main” routing tableThis will also allow proper communicationbetween locally and remotely connected clients MikroTik 201220

Remote ConnectionsIn the case when a connection is initiated from apublic interface it is necessary to ensure thatthese connections will be replied via the sameinterface (from the same public IP)First we need to capture these connections (youcan ether use default connection mark “nomark” or connection state “new” here) MikroTik 201221

Custom Policy RoutingNow we need to create a default route for everyrouting table (or else it will be resolved by mainrouting table)Let's create a jump rule to your custom policyrouting here MikroTik 201222

Mark RoutingMark routing rules in mangle chain “output” willensure that router itself is reachable via bothpublic IP addressesMark routing rules in mangle chain “prerouting”will ensure your desired load balancing MikroTik 201223

Mangle configuration MikroTik 201224

Custom Policy RoutingThere is no best way that we can suggest forload balancing, you can either:Balance based on client IP address (address list)Balance based on traffic type (p2p, layer-7, protocol,port)Use automatic balancing (PCC)We do not suggest to use “nth” for policy routingof typical user traffic. MikroTik 201225

Per-address-pair Load BalancingIn many situations communication between twohosts consist of more than one simultaneousconnection.If those connections are taking different routingpaths they might have different latency, droprate, fragmentation or source address (NAT) –this way making multi-connectioncommunications impossible.That is why instead of per-connection loadbalancing we should think about per-addresspair load balancing MikroTik 201226

Per Connection ClassifierPCC is a firewall matcher that allows you todivide traffic into equal streams with ability tokeep packets with specific set of options in oneparticular streamYou can specify set of options from src-address,src-port, dst-address, dst-portMore info at:http://wiki.mikrotik.com/wiki/PCC MikroTik 201227

PCC ConfigurationWe just need to add 2 rules to our“policy routing” chain to ensure automatic peraddress-pair load balancing MikroTik 201228

Usual ProblemsBe careful about using “no-mark” connectionmark if you have other mangle configuration in adifferent chainISP specified DNS servers might block requestsfrom non-ISP public IPs, so we suggest you usepublic (ISP independent) DNS servers.If you would like to ensure fail-over – enable“check-gateway” option in all default routes. MikroTik 201229

What about bandwidth basedLoad-Balancing? MikroTik 201230

Traffic EngineeringTE is one of MPLS features that allow toestablish unidirectional label switching pathsTE is based on RSVP (Resource ReSerVationProtocol) RFC 3209 that adds support forexplicit route and label exchangeTE tunnels are similar to LDP, but withadditional features:Usage of either full or partial explicit routesConstraint (such as bandwidth and link properties)based LSP (Label Switched Path) establishment MikroTik 201231

How Does Constraints Work?Constraints are set by user and does notnecessarily reflect actual bandwidthConstraints can be set for:bandwidth of link participating in a RSVP TEnetworkbandwidth reserved for tunnelSo, at any moment in time, the bandwidthavailable on TE link is bandwidth configured forlink minus sum of all reservations made on thelink (not physically available bandwidth) MikroTik 201232

TE Tunnel EstablishmentTE tunnels can be established:along the current routing path (no additionalconfiguration required)along a statically configured explicit path (it isnecessary to manually input path)CSPF (Constrained Shortest Path First) - Thisoption needs assistance from IGP routing protocol(such as OSPF) to distribute bandwidth informationthroughout the network. MikroTik 201233

Network LayoutEach router is connected to a neighbouringrouter using /30 network and each of them haveunique Loopback address form 10.255.0.xnetwork. Loopback addresses will be used astunnel source and destination. MikroTik 201234

Network Layout MikroTik 201235

Loopback and CSPFLoopback addresses need to be reachable fromwhole network – we will use OSPF to distributethat informationAlso OSPF can help us to distribute TEreservations for CSPF MikroTik 201236

Resource ReservationLets set up TE resource for every interface onwhich we might want to run TE tunnel.Configuration on all the routers are the same:Note that at this point this does not representhow much bandwidth will actually flow throughthe interface MikroTik 201237

First Task MikroTik 201238

TE tunnel setupWe will use static path configuration asprimary, and dynamic (CSPF) as secondarypath if primary fails MikroTik 201239

TE Tunnel Monitoring MikroTik 201240

TE Tunnel MonitoringIf multiple tunnels are created and all thebandwidth on that particular interface is used,then the tunnel will try to look for different path. MikroTik 201241

Route traffic over TETo route LAN traffic over a TE tunnel we willassign address 10.99.99.1/30 and10.99.99.2/30 to each tunnel end. MikroTik 201242

Automatic FailoverBy default the tunnel will try to switch back tothe primary path every minute. This setting canbe changed with primary-retry-intervalparameter. MikroTik 201243

Additional Tunnels MikroTik 201244

Additional Tunnels MikroTik 201245

Good luck!http://wiki.mikrotik.com/wiki/Manual:Simple TEhttp://wiki.mikrotik.com/wiki/Manual:TE S/Overview MikroTik 201246

MikroTik 2012 MikroTik RouterOS Workshop Load Balancing Best Practice Warsaw MUM Europe 2012