Network Automation At Scale - NANOG Archive
Network automation at scaleUp and running in 60 minutesMircea UlinicCloudflare, LondonNANOG 69 Washington, D.C.February 20171
Why us? How big? Four million zones/domainsAuthoritative for 40% of Alexa top 1 million43 billion DNS queries/day Second only to Verisign100 anycast locations globally 50 countries (and growing)Many hundreds of network devices2
To automate, I have to learn Python or anotherprogramming language.3
NOR!GTo automate, I have to learn Python or anotherprogramming language.W4
Prerequisites No programming skills required (but verywelcome)! Basic system ops Networking (of course) Basic YAML & Jinja understanding(6 simple rules is all you need for the beginning)5
Agenda Meet the toolsInstall the toolsConfigure Vagrant VMs (demo only)Configure SaltStackCLI syntaxConfiguration managementAdvanced topics6
Meet the ToolsLive setup Access to a remote serverOR Vagrant VM(s) from your favouritevendor(s)NOTE: Vagrant is used for demo only!The power of SaltStack can be seen when managing high number of realnetwork devices!7
Meet the ToolsWhy Salt? Very scalableConcurrencyEasily configurable & customizableConfig verification & enforcementPeriodically collect statisticsNative caching and drivers for useful tools8
Meet the ToolsOrchestration vs. AutomationCC BY 2.0 https://flic.kr/p/5EQe2d9
Meet the ToolsWhy Salt?“In SaltStack, speed isn’t a byproduct, it is a design goal. SaltStack was createdas an extremely fast, lightweight communication bus to provide the foundationfor a remote execution engine.SaltStack now provides orchestration, configuration management, eventreactors, cloud provisioning, and more, all built around the SaltStackhigh-speed communication bus. cross-vendor network automation from 2016.11 d/speed.html10
Meet the ToolsWhy NAPALM?(Network Automation and Programmability Abstraction Layer with Multivendor 11
12
NAPALM integrated in ics/releases/2016.11.0.html13
Install the toolsInstall NAPALM pip install napalmSee Complete installation notes14
Install the toolsInstall SaltStack sudo apt-get install salt-master sudo apt-get install salt-proxySee Complete installation notes15
Configure VagrantThis assumes Vagrant and VirtualBox arealready installedVagrantfile examples:What I useSomething simplerNOTE: skip this section if you are running in a real network environment (preferable)16
Configure VagrantDownload vEOSGo to Arista software download (accountrequired)Select any .box file, but makesure that VEOS BOX matchesthe name in the Vagrantfile.17
Configure VagrantDownload vSRX vagrant box add juniper/ffp-12.1X47-D20.7-packetmode box: Loading metadata for box 'juniper/ffp-12.1X47-D20.7-packetmode'box: URL: -packetmodeThis box can work with multiple providers! The providers that itcan work with are listed below. Please review the list and choosethe provider you will be working with.1) virtualbox2) vmware desktopEnter your choice: 1 box: Adding box 'juniper/ffp-12.1X47-D20.7-packetmode' (v0.5.0) for provider: virtualboxbox: oviders/virtualbox.box box: Successfully added box 'juniper/ffp-12.1X47-D20.7-packetmode' (v0.5.0) for 'virtualbox'!18
Configure VagrantStart Vagrant boxes vagrant up vsrxBringing machine 'vsrx' up with 'virtualbox' provider. vsrx: Setting the name of the VM: mirucha vsrx 1483551699725 41640 vsrx: Clearing any previously set network interfaces. vsrx: Preparing network interfaces based on configuration.vsrx: Adapter 1: natvsrx: Adapter 2: intnetvsrx: Adapter 3: intnetvsrx: Adapter 4: intnetvsrx: Adapter 5: intnet vsrx: Forwarding ports.vsrx: 22 (guest) 12202 (host) (adapter 1)vsrx: 830 (guest) 12830 (host) (adapter 1)vsrx: 80 (guest) 12280 (host) (adapter 1) vsrx: Booting VM. vsrx: Waiting for machine to boot. This may take a few minutes.vsrx: SSH address: 127.0.0.1:12202vsrx: SSH username: vagrantvsrx: SSH auth method: private keyvsrx:19
Configure SaltStackNew to Salt?PillarFree-form data that can be used to organize configurationvalues or manage sensitive data, e.g.: interface details,NTP peers, BGP config.written by the user, generally one file per deviceGrainsdata collected from the device, e.g.: device model, vendor,uptime, serial number etc.Salt handles this, you don’t need to do anythingSalt in 10 minutes: als/walkthrough.html20
Configure SaltStackMaster config/etc/salt/masterfile roots:base:- /etc/salt/statespillar roots:base:Environment nameUseful to have differentenvironments: prod, qa,develop etc.- /etc/salt/pillarFor the beginning, let’s focus only on file roots and pillar roots. The others settings are moreadvanced features: tion/master.htmlComplete salt master config file21
Configure SaltStackProxy config/etc/salt/proxymaster: localhostpki dir: /etc/salt/pki/proxycachedir: /var/cache/salt/proxymultiprocessing: FalseVery important!mine enabled: TrueMore about proxy minions: inion/index.html22
Configure SaltStackDevice pillarUnder the pillar roots directory (as configured in yproxy:Mandproxytype: napalmdriver: junoshost: hostname or ip addressusername: my usernamepasswd: my passwordChoose between: juneos, ios,iosxr, nxo os,s, etc.See the complete list.Complete documentation at: l/salt.proxy.napalm.html23
Configure SaltStackThe top fileUnder the pillar roots directory (as configured in ion IDdevice1:Environment nameUseful to have differentenvs: prod, qa, developetc.- device1This is how the device will beidentified from now on.It can be anything, does notneed to match with the .slsfile or the hostname.device2:.sls file to be included- device2Specify the name of the .slsfile descriptor (earlierdefined).Do NOT include the .slsextension.24
Configure SaltStackmaster systemd file Unit]Description Salt MasterRequires network.targetAfter network.target[Service]Type forkingPIDFile /var/run/salt-master.pid# ***NOTE*** the virtualenv here!Your location may vary!ExecStart /usr/bin/salt-master -dRestart on-failureRestartSec 15[Install]WantedBy multi-user.target25
Configure SaltStackproxy systemd file Unit]Description Salt proxy minionAfter network.target[Service]Type simple# ***NOTE*** the virtualenv here!Your location may vary!ExecStart /usr/bin/salt-proxy -l debug --proxyid %IUser rootGroup rootRestart alwaysRestartPreventExitStatus SIGHUPRestartSec 5[Install]WantedBy default.target26
Configure SaltStackStart the salt-master With systemd: sudo systemctl start salt-master Without systemd:Start as daemon sudo salt-master -d27
Configure SaltStackStart the salt-proxy processes With systemd: sudo systemctl start salt-proxy@device1 sudo systemctl start salt-proxy@device2minion ID Without systemd:As configured inthe top file. sudo salt-proxy -d --proxyid device1 sudo salt-proxy -d --proxyid device228
Configure SaltStackAccept the proxies connection to the masterFor each device, accept the minion key:minion ID sudo salt-key -a device1The following keys are going to be accepted:Unaccepted Keys:As configured inthe top file.device1Proceed? [n/Y] yKey for minion device1 accepted.This is due to security reasons.More about salt-key: key.htmlNOTE: Accepting the minion keys can be automated as well.29
Done!You are now ready to automate your network!30
Salt CLI syntaxSelecting the devices we need to run the command.Targeting can be s/targeting/ sudo salt target function [ arguments ]Function name, as specified in themodule documentation.For example if we need BGP-relatedcommands, we’ll look at the BGPmodule.Other examples: dnsutil.A, net.arp,net.lldp, net.traceroute etc.Function arguments, as specified in themodule documentation.Some functions do not require anyarguments.31
Salt CLI syntaxExamples sudo salt 'edge*' net.traceroute 8.8.8.8# execute traceroute on all devices whose minion ID starts with ‘edge’ sudo salt -N NA transit.disable cogent‘NA’ is a nodegroup:# disable Cogent in North-America sudo salt -G 'os:junos' net.cli “show version”# execute ‘show version’ on all devices running argeting/nodegroups.html sudo salt -C 'edge* and G@os:iosxr and G@version:6.0.2' net.arp# get the ARP tables from devices whose ID starts with edge*, running IOS-XR 6.0.2 sudo salt -G 'model:MX480' probes.results# retrieve the results of the RPM probes from all Juniper MX480 routers32
sudo salt edge01.iad01 net.arpedge01.iad01:---------out: ---------age:129.0interface:ae2.100Salt CLI syntaxOutput exampleip:10.0.0.1mac:00:0f:53:36:e4:50 ---------age:Default output style: :1d:70:83:40:c033
Salt CLI syntaxOutputters salt --out json edge01.iad01 net.arp salt --out yaml edge01.iad01 net.arp[edge01.iad01:comment: ''{"interface": "ae2.100",out:"ip": "10.0.0.1",- age: 129.0"mac": "00:0f:53:36:e4:50","age": 129.0},{"interface": "xe-0/0/3.0",Using the --outoptionalargument, onecan select theoutput format.interface: ae2.100ip: 10.0.0.1mac: 00:0f:53:36:e4:50- age: 1101.0interface: xe-0/0/3.0"ip": "10.0.0.2",ip: 10.0.0.2"mac": "00:1d:70:83:40:c0",mac: 00:1d:70:83:40:c0"age": 1101.0},Other outputters: ll/index.html34
ConfigurationmanagementLoad staticconfig sudo salt -G 'vendor:arista' net.load config text 'ntp server 172.17.17.1'edge01.bjm01:---------already configured:Falsecomment:diff:Match all Aristadevices fromthe network.@@ -42,6 42,7 @@ntp server 10.10.10.1ntp server 10.10.10.2ntp server 10.10.10.3 ntp server 172.17.17.1ntp serve allConfig diff!result:Trueedge01.pos01:No changesrequired on thisdevice.---------already configured:Truecomment:diff:result:True35
ConfigurationmanagementLoad staticconfig: dry-run sudo salt edge01.bjm01 net.load config text 'ntp server 172.17.17.1' test Trueedge01.bjm01:---------already configured:Falsecomment:Configuration discarded.Dry-run modediff:@@ -42,6 42,7 @@ntp server 10.10.10.1ntp server 10.10.10.2Changes arediscarded.ntp server 10.10.10.3 ntp server 172.17.17.1ntp serve all!result:True36
ConfigurationmanagementLoad staticconfigLoading static config(more changes) sudo salt edge01.bjm01 net.load config /home/mircea/arista ntp servers.cfg test Trueedge01.bjm01:---------already configured:Falsecomment:Configuration discarded.diff:Absolute path@@ -42,6 42,10 @@ntp server 10.10.10.2ntp server 10.10.10.3 ntp server 172.17.17.1 ntp server 172.17.17.2 ntp server 172.17.17.3 cat /home/mircea/arista ntp servers.cfg ntp server 172.17.17.4ntp server 172.17.17.1ntp serve allntp server 172.17.17.2!ntp server 172.17.17.3ntp server 172.17.17.4result:True37
Configuration managementInline Templating sudo salt edge01.bjm01 net.load template set hostname template source 'hostname {{ host name }}' host name 'arista.lab’edge01.bjm01:----------Observe the functionname is:net.load templatealready configured:Falsecomment:Inline templateTemplate vardiff:@@ -35,7 35,7 @@logging console emergencieslogging host 192.168.0.1!-hostname edge01.bjm01 hostname arista.lab!result:TrueNOTE: the template is evaluated on the minion38
Configuration managementGrains inside the templates sudo salt edge01.bjm01 net.load template set hostname template source 'hostname {{ grains.model }}.lab'edge01.bjm01:---------already configured:Falsecomment:Router modelis collectedfrom the grainsdiff:@@ -35,7 35,7 @@logging console emergencieslogging host 192.168.0.1!-hostname edge01.bjm01 hostname DCS-7280SR-48C6-M-R.lab!result:True39
Configuration managementCross vendor templating (1)/home/mircea/example.jinja{%- set router vendor grains.vendor -%}{%- set hostname pillar.proxy.host -%}{%- if router vendor lower 'juniper' %}system {Hostname alreadyspecified in thepillar.Get the devicevendor from thegrainshost-name {{hostname}}.lab;}{%- elif router vendor lower in ['cisco', 'arista'] %}{# both Cisco and Arista have the same syntax for hostname #}hostname {{hostname}}.lab{%- endif %}40
Configuration managementCross vendor templating (2) sudo salt '*' net.load template dge01.flw01:already configured:Falsecomment:diff:----------Arista deviceJuniper devicealready configured:Falsecomment:@@ -35,7 35,7 @@diff:logging console emergencies[edit system]logging host 192.168.0.1-host-name edge01.flw01;! host-name edge01.flw01.lab;-hostname edge01.bjm01 hostname edge01.bjm01.labresult:True!result:TrueMany vendors, one simple template!41
Configuration managementDebug mode sudo salt edge01.flw01 net.load template /home/mircea/example.jinja debug Trueedge01.flw01:---------already configured:Absolute pathDebug modeFalsecomment:diff:[edit system]-host-name edge01.flw01; host-name edge01.flw01.lab;loaded config:system {host-name edge01.flw01.lab;}result:TrueThe result of template rendering.Not necessarily equal to the diff.Note: Jinja is painful to debug.This option is very helpful.See more debugging tools42
Configuration managementThe right way to specify the template sourceedge01.flw01:} sudo salt edge01.flw01 net.load template salt://templates/example.jinja debug True---------already configured:Falsecomment:diff:Translated to file roots,as specified in the master config file - see slide #21.[edit system]-host-name edge01.flw01; host-name edge01.flw01.lab;loaded config:E.g.: if file roots is configured as /etc/salt/states/, thephysical location of the template is/etc/salt/states/templates/example.jinjasystem {host-name edge01.flw01.lab;}result:TrueNote:Under file roots, one can also add:/etc/salt/templates, define the template file under thepath: /etc/salt/templates/example.jinja and call using:salt://example.jinja43
Configuration managementRemote templatesYes, they can also be elsewhere.Available options: salt://, ftp://, http://, https://,version control, cloud storage providers etc. sudo salt -G 'os:ios' net.load template http://bit.ly/2gKOj20 peers "['172.17.17.1', '172.17.17.2']"Matches alldevices runningIOSLoads external templatefrom http://bit.ly/2gKOj20which shortens the link tothe NAPALM native template for IOS.44
Configuration managementAdvanced templating: reusing existing data (1){%- set arp output salt.net.arp() -%}{%- set arp table arp output['out'] -%}/etc/salt/templates/arp example.jinja{%- if grains.os lower 'iosxr' %} {# if the device is a Cisco IOS-XR #}{%- for arp entry in arp table %}arp {{ arp entry['ip'] }} {{ arp entry['mac'] }} arpa{%- endfor -%}{%- elif grains.vendor lower 'juniper' %} {# or if the device is a Juniper #}Retrieving the ARPtable using thenet.arp function.interfaces {{%- for arp entry in arp table %}{{ arp entry['interface'] }} {family inet {address {{ arp entry['ip'] }} {arp {{ arp entry['ip'] }} mac {{ arp entry['mac'] }};}}}{%- endfor %}}{%- endif %}45
Configuration managementAdvanced templating: reusing existing data (1) sudo salt edge01.flw01 net.load template salt://arp example.jinjaedge01.flw01:---------already configured:Falsecomment:diff:[edit interfaces xe-0/0/0 unit 0 family inet] address 10.10.2.2/32 { arp 10.10.2.2 mac 0c:86:10:f6:7c:a6;}[edit interfaces ae1 unit 1234] family inet { address 10.10.1.1/32 { arp 10.10.1.1 mac 9c:8e:99:15:13:b3; }}result:True46
Configuration managementAdvanced templating: reusing existing data (2)/etc/salt/templates/route example.jinja{%- set route output salt.route.show('0.0.0.0/0', 'static') -%}{%- set default route route output['out'] -%}{%- if not default route -%} {# if no default route found in the table #}Retrieving the staticroute data using theroute.show function.{%- if grains.vendor lower 'juniper' -%}routing-options {static {route 0.0.0.0/0 next-hop {{ pillar.default route nh }};}This requiresappending a new line inthe device pillar:}{%- elif grains.os lower 'iosxr' -%}default route nh: 1.2.3.4router static address-family ipv4 unicast 0.0.0.0/0 {{ pillar.default route nh }}{%- endif %}{%- endif -%}47
Configuration managementAdvanced templating: reusing existing data (2) sudo salt 'edge01.oua01' net.load template salt://route example.jinja debug Trueedge01.oua01:---------already configured:Falsecomment:diff:-- @@ -3497,6 3497,7 @@!router staticaddress-family ipv4 unicast 0.0.0.0/0 1.2.3.4172.17.17.0/24 Null0 tag 100loaded config:router static address-family ipv4 unicast 0.0.0.0/0 1.2.3.4result:True48
Homework: other simple examples Using postgres.psql query populate a table in a Postgres database withthe network interfaces details (retrieved using net.interfaces)Using bgp.neighbors remove from the BGP config neighbors in ActivestateUsing ntp.stats, remove unsynchronised NTP peersUsing net.environment, push high temperature notifications in SlackThe list can be nearly infinite - depends only on your own use case.There are thousands of functions already f/modules/all/index.htmlNote: the examples above are implemented more elegant using states, beacons, reactors, etc.49
Advanced topicsStates, schedulers, reactors, beacons, APIThese are advanced topics, that require the user to read carefully thedocumentation.Using these types of modules, one can control the configuration based onevents, either external or internal, e.g.:-BGP neighbor down triggers a BGP configuration changeGit pull-request merged triggers configuration updateHigh temperature alert triggers a notification post in a Slack channelChatOpsetc.50
Advanced topicsStateA state ensures that on the devices you have configured what you expect tobe. What’s not defined in the pillar, it will be removed; what’s not on thedevice, but it’s defined in the pillar, will be added.Integrated states: netntpnetsnmpnetusersprobesnetconfig (very important; will be added in the next release: Nitrogen)51
Advanced topicsState example: update NTP peers (1)Append directly these linesin the device pillar, or definein external file and include:/etc/salt/pillar/ntp oxy:- 10.10.1.1proxytype: napalm- 10.10.2.2driver: junosntp.servers:host: hostname or ip address- 172.17.17.1username: my username- 172.17.19.1passwd: my passwordBetter to use the include, asmultiple devices can havethe same NTP peers etc.include:- ntp configWhen including, strip the .slsextension!52
Advanced topicsState example: update NTP peers (1)As configured under file roots/etc/salt/states/router/ntp.sls{% set ntp peers pillar.get('ntp.peers', []) -%}{% set ntp servers pillar.get('ntp.servers', []) -%}Take the NTP peers/serversfrom the pillar (earlier defined)update my ntp config:netntp.managed:- peers: {{ ntp peers json() }}- servers: {{ ntp servers json() }}Pass them as state argumentsBest practice:Although not mandatory, usethe json() filter to explicitlyserialize objects.This is the state virtualname, more s/all/salt.states.netntp.html53
Advanced topicsState example: update NTP peers (3)This file is - ntpInclude the earlier definedstate SLS file. sudo
Network automation at scale Mircea Ulinic Cloudflare, London NANOG 69 Washington, D.C. February 2017 1 Up and running in 60 minutes
Automation Journey! Reporting Most network engineers begin their automation journey by producing some simple reporting software. It is low-risk, has a positive useful impact, and a good introduction to network scripting and the many libraries that support network automation. 2
Feb 20, 2018 · – Network Automation Lead – owns the automation efforts and works to remove roadblocks with other departments, vendors, etc. – Automation Designer – defines the work to be done, tools to use, workflow/steps of automation, and acts as technical lead – Engineer – works w
you can control how automation is deployed, and gain auditable knowledge about automation sources and outcomes. You can also use Red Hat Ansible Network Automation, a bundled offering tailored for network automation tasks. Read the Network automation for everyone e-book to learn more about Red Hat Ansible Network Automation. HOW TO USE THIS E-BOOK
CCC-466/SCALE 3 in 1985 CCC-725/SCALE 5 in 2004 CCC-545/SCALE 4.0 in 1990 CCC-732/SCALE 5.1 in 2006 SCALE 4.1 in 1992 CCC-750/SCALE 6.0 in 2009 SCALE 4.2 in 1994 CCC-785/SCALE 6.1 in 2011 SCALE 4.3 in 1995 CCC-834/SCALE 6.2 in 2016 The SCALE team is thankful for 40 years of sustaining support from NRC
programmable logic controller, is important for industrial engineer. Factory automation mainly covers; Machine level automation, Production line or work cell automation, Shop floor automation, and Plant level automation. The present manual focus on the 1st level of factory automation e.g. machine automation level. It provides an introduction .
Network Automation 101 Ivan Pepelnjak (ip@ipSpace.net) Network Architect . Lack of programming skills Lack of reliable automation tools and programmatic interfaces . NetworkAutomation 101 Network Programmability 101 Network Automation Tools Network Automation Use Cases.
Svstem Amounts of AaCl Treated Location Scale ratio Lab Scale B en&-Scale 28.64 grams 860 grams B-241 B-161 1 30 Pilot-Plant 12500 grams MWMF 435 Table 2 indicates that scale up ratios 30 from lab-scale to bench scale and 14.5 from bench scale to MWMW pilot scale. A successful operation of the bench scale unit would provide important design .
The SBSS-prepared A02 and A0B MILS transactions carry the expanded length descriptive data, which could contain various types of information for part-numbered requisitions in rp 67-80, and requires mapping to the DLMS transaction. This information is common to the YRZ exception data used by DLA, and so can be mapped to the generic note field as specified above. b. DLMS Field Length .
