ACI Anywhere: Extending The ACI Fabric - Cisco
ACI Anywhere: Extending the ACI FabricMax ArdicaPrincipal Engineer – Cisco INSBU2
Agenda§ ACI Network and Policy Domain Evolution§ ACI Multi-Pod§ ACI Multi-Site§ ACI Remote Physical Leaf§ ACI Remote Virtual Leaf (vPod)§ ACI Extensions to Multi-Cloud§ Conclusions and Q&A
ACI Network and Policy DomainEvolution
Introducing: Application Centric Infrastructure iceFilterAPICACI FabricApplication PolicyInfrastructure ControllerIntegrated GBP VXLAN Overlay19 - 21 March 2018 Cisco Connect Rovinj, Croatia5
Cisco ACI: Industry Leader4,800 46 %65 ACI CustomersACI Attach RateEcosystem PartnersEcosystem Partners19 - 21 March 2018 Cisco Connect Rovinj, Croatia6
Cisco ACIFabric and Policy Domain EvolutionACI Multi-SiteACI Single Pod FabricACI Multi-CloudIPFabric ‘A’Fabric ‘n’MP-BGP - EVPN ACI 2.0 - Multiple Networks(Pods) in a single AvailabilityZone (Fabric)ACI 1.0 Leaf/Spine SinglePod FabricACI Multi-Pod FabricPod ‘A’IPNPod ‘n’ACI 3.1/3.2 - Remote Leafand vPod extends anAvailability Zone (Fabric) toremote locationsACI 3.0 – Multiple AvailabilityZones (Fabrics) in a SingleRegion ’and’ Multi-RegionPolicy ManagementACI Remote LeafFuture – ACI Extensionsto Multi-CloudMP-BGP - EVPN APIC Cluster19 - 21 March 2018 Cisco Connect Rovinj, Croatia7
Regions and Availability ZonesOpenStack and AWS DefinitionsOpenStack Regions - Each Region has its own full OpenStackdeployment, including its own API endpoints, networksand compute resources Availability Zones - Inside a Region, compute nodes canbe logically grouped into Availability Zones, when launchingnew VM instance, we can specify AZ or even a specificnode in a AZ to run the VM instanceAmazon Web Services Regions – Separate large geographical areas, eachcomposed of multiple, isolated locations known asAvailability Zones Availability Zones - Distinct locations within a regionthat are engineered to be isolated from failures in otherAvailability Zones and provide inexpensive, low latencynetwork connectivity to other Availability Zones in thesame region19 - 21 March 2018 Cisco Connect Rovinj, Croatia8
Typical RequirementCreation of Two Independent Fabrics/AZsFabric ‘A’ (AZ 1)Fabric ‘B’ (AZ 2)Applicationworkloadsdeployed acrossavailability zones19 - 21 March 2018 Cisco Connect Rovinj, Croatia9
Typical RequirementCreation of Two Independent Fabrics/AZsMulti-Pod Fabric ‘A’ (AZ 1)‘Classic’ Active/ActivePod ‘2.A’Pod ‘1.A’ACI Multi-SiteMulti-Pod Fabric ‘B’ (AZ 2)‘Classic’ Active/ActiveApplicationPod ‘2.B’Pod ‘1.B’workloadsdeployed acrossavailability zones19 - 21 March 2018 Cisco Connect Rovinj, Croatia10
ACI Multi-Pod
ACI Multi-PodOverviewPod ‘A’VXLANInter-Pod NetworkPod ‘n’MP-BGP - EVPN APIC ClusterIS-IS, COOP, MP-BGPIS-IS, COOP, MP-BGPAvailability Zone Multiple ACI Pods connected by an IP Inter-PodL3 network, each Pod consists of leaf and spinenodes Managed by a single APIC Cluster Single Management and Policy Domain19 - 21 March 2018 Cisco Connect Forwarding control plane (IS-IS, COOP)fault isolation Data Plane VXLAN encapsulation betweenPods Rovinj,End-to-endpolicy enforcementCroatiaBRKACI-200312
Single Availability Zone with Tenant IsolationIsolation for ‘Virtual Network Zone and Application’ ChangesInter-Pod NetworkACI Multi-PodFabricAPIC ClusterTenant ‘Prod’ Configuration/Change DomainTenant ‘Dev’ Configuration/Change Domain The ACI ‘Tenant’ construct provide a domain of application and associated virtualnetwork policy change Domain of operational change for an application (e.g. production vs. test)19 - 21 March 2018 Cisco Connect Rovinj, Croatia13
ACI Multi-PodSupported TopologiesTwo DC sites directly connectedIntra-DC1G/10G/40G/100GPOD 110G*/40G/100G10G*/40G/100GPOD nPOD 110G*/40G/100GAPIC ClusterAPIC ClusterMultiple sites interconnected by ageneric L3 network3 (or more) DC Sites directly connected1G/10G/40G/100GDark fiber/DWDMPOD 2(up to 50** msec RTT) POD 1 10G*/40G/100G10G*/40G/100GDark fiber/DWDM10G*/40G/100GPOD 210G*/40G/100G10G*/40G/100G(up to 50 msec RTT)L310G*/40G/100GPOD 310G*/40G/100G- 21QSAMarchadapters2018 CiscoConnectRovinj, Croatia* 10G only 19withon EX/FXand 9364Cspines(up to 50msec RTT)10G*/40G/100G** 50 msec support added in SW release 2.3(1)14
ACI Multi-Site
ACI Multi-SiteOverviewVXLANIP NetworkMP-BGP - EVPNMulti-Site OrchestratorSite 1RESTAvailability Zone ‘A’ APISite 2GUIAvailability Zone ‘B’Separate ACI Fabrics with independent APIC clusters ACI Multi-Site Orchestrator pushes cross-fabric configuration to multiple APIC clusters providingscoping of all configuration changes 19 - 21 March 2018 Cisco ConnectMP-BGP EVPN control plane between sitesData Plane VXLAN encapsulation acrosssitesEnd-to-end policy definition and enforcement Rovinj, Croatia16
ACI Multi-SiteMain Use CasesScale-Up Model to Build a LargeIntra-DC Network19 - 21 March 2018Data Center Interconnect (DCI) Cisco Connect Rovinj, Croatia17
ACI Multi-SiteNetwork and Identity Extended between FabricsNetwork information carried acrossFabrics (Availability Zones)Identity information carried acrossFabrics (Availability Zones)VTEP IPVNIDTenant PacketClass-IDIP NetworkNo Multicast Requirement inBackbone, Head-EndReplication (HER) for anyLayer 2 BUM traffic)MP-BGP - EVPN19 - 21 March 2018 Cisco Connect Rovinj, Croatia18
ACI Multi-SiteNamespace NormalizationVNID à 16678781Class-ID: 49153IP NetworkVNID à 16678781Class-ID: 49153Translation of Class-ID, VNID(scoping of name spaces)Spine Translation TableMP-BGP - EVPNVNID à 16547722Class-ID: 32770VNID à 16678781Class-ID: 49153EP1EPGSite 1CRem. SiteLocal P2EPGSite 2EP2EPGLeaf to Leaf VTEP, Class-ID is local to the FabricLeaf to Leaf VTEP, Class-ID is local to the Fabric Maintain separate name spaces with ID translation performed on the spine nodes VNIDClass-IDRequires specific HW on the spine to support for this functionality Multi-Site Orchestrator instructs local APIC to program translation tables on spinesVNIDVNIDClass-IDClass-IDTenant PacketTenant Packet19 - 21 March 2018 Cisco Connect Rovinj, CroatiaTenant Packet19
ACI Multi-SiteHardware Requirements§ Support all ACI leaf switches (1st Generation, -EX and -FX)§ Only –EX spine (or newer) toconnect to the inter-site networkCan have only a subsetof spines connecting tothe IP networkIP Network§ New 9364C non modular spine(64x40G/100G ports) supported forMulti-Site from ACI 3.1 release1st Gen 1st Gen-EX-EX§ 1st generation spines (including 9336PQ)not supported§ Can still leverage those for intra-site leafto leaf communication19 - 21 March 2018 Cisco Connect Rovinj, Croatia20
ACI Multi-Site Networking OptionsPer Bridge Domain BehaviorLayer 3 only across sites1Site1Site2ISNSiteSite22§Bridge Domains and subnets notextended across Sites§Same IP subnet defined in separateSites§Layer 3 Intra-VRF or Inter-VRFcommunication (shared servicesacross VRFs/Tenants)§Support for IP Mobility (‘cold’ and‘live’* VM migration) and intrasubnet communication across sites§MSO GUI(BD)32ISNISNSite1Layer 2 adjacency across SitesIP Mobility without BUM floodingNo Layer 2 BUM flooding acrosssites§Interconnecting separate sites forfault containment and scalabilityreasons§Layer 2 domains stretched acrossSites, support for ‘live’* VMmigration and application clustering§Layer 2 BUM flooding acrosssitesMSO GUI(BD)MSO GUI(BD)19 - 21 March 2018Site2Site1 Cisco *’Live’Connect migration Rovinj, Croatiaofficiallysupported from ACI release 3.221
ACI Multi-SiteCloudSec Encryption for VXLAN TrafficEncrypted Fabric to Fabric TrafficVTEP Information Clear Text[ GCM-AES-128 (32-bit PN), GCM--AES-256(32-bit PN), GCM-AES-128-XPN (64-bit PN),GCM-AES-256-XPN (64-bit PN)])VTEP IPMACSECTenant PacketVXLANInter-Site NetworkMP-BGP - EVPNSupport planned for a future ACI release for FX line cards and 9364C platform19 - 21 March 2018 Cisco Connect Rovinj, Croatia22
ACI Multi-SiteMulti-Site Orchestrator (MSO) Micro-services architecture§RESTGUIAPI§ACI Multi-SiteVMVM VMOOB Mgmt connectivity to the APIC clustersdeployed in separate sites§Hypervisor Site 2§§Site n§§19 - 21 March 2018Support for 500 msec to 1 sec RTTMain functions offered by MSO: .Site 1Multiple MSO nodes are created and runconcurrently (active/active)vSphere VM only form factor initially (physicalappliance planned for a future ACI release) Cisco Connect Monitoring the health-state of the different ACI SitesProvisioning of day-0 configuration to establishinter-site EVPN control planeDefining and provisioning policies across sitesDay-2 operation functionalitiesRovinj, Croatia23
ACI Multi-SiteMSO Schema and Templates Template APIC policy definition(App & Network)Template is the scope/granularity of whatcan be pushed to sitesTemplate is associated to all managed sitesor to a subset of sitesSchema container of Templates sharing acommon use-caseScope of change: policies in differenttemplates can be pushed to separate sitesat different times19 - 21 March P2EPGSITELOCALSite 2Site 1EFFECTIVEPOLICY Cisco Connect Rovinj, CroatiaEFFECTIVEPOLICY24
ACI Multi-SiteDay-2 Operations: Full-Stack Consistency CheckerACI 3.2 Release Multi-Site Infra: Unicast, Multicast, BGP TEPsand Tunnel stateMulti-Site Tenant and EPG granularity:§Inspect and validate full-stack programming: MSC,APICs and Spine translations§Validate the consistency of local and remote intersite EPGs, BD, VRF, External EPG, policies, etc.§Root cause configuration programming issueswithout calling TACMP-BGP EVPNSpinesVXLANSpines 19 - 21 March 2018GUI and APIs will both be supported Cisco Connect Rovinj, Croatia25
ACI Remote Physical Leaf
ACI Remote Physical LeafBusiness Value and Use CasesExtending the ACI policy model outside the main datacenter to remotesites distributed over IP Backbone (Satellites DCs, CoLo locations, etc.)Extending ACI fabric policy and L2/L3 connectivity to a small DR sitewithout requiring the deployment of a full-blown ACI FabricCentralized Policy Management and Control Plane for remote locationsSmall form factor solution at locations with space constraints19 - 21 March 2018 Cisco Connect Rovinj, Croatia27
ACI Remote Physical LeafConceptual ArchitectureA Remote Leaf ‘Site’ getsassociated with the Spines ofone specific Pod in Main DCAPIC and Spine Nodes (Proxyfunction) remain at primary Pod(s)IP WANIPNL2 / L3IPN Requirements (3.1)§§§§150 msec maximum RTT500 Mbps minimum BW1600B minimum MTUNo PIM-Bidir tureACI Main DC19 - 21 March 2018 Cisco Connect Remote Leaf Site: a pair of Nexus 9300 nodesconnected to a L3 Network via uplink ports andfully managed by a centralized APIC clusterRovinj, Croatia28
ACI Remote Physical LeafHardware/Software SupportAll hardware from -EX onwards is required for remote leaf nodesand the spines to which they get associatedACI Main PodRemote Leaf NodesSupported SpinesSupported LeafFixed N9364CModular N9732C-EX N9736C-FX 19 - 21 March 2018 Cisco Connect 3108TC-FXN9K-C9348GC-FXPRovinj, Croatia29
ACI Remote Physical LeafFunctionalities and ScaleACI Main DCOnly 2 RLsper ‘RL Site’IP WANIPNRemote Leaf SiteL3Out§All major features supported on RL nodes§Scalability values are the same as for localleaf deployments§vSwitchHypervisorService Graph(unmanaged mode) withPBR40 RL (20 sites, 2 RL per site)19 - 21 March 2018BareMetal Cisco Connect Rovinj, Croatia30
ACI Remote Physical LeafAutomatic RL DiscoveryOSPF can beredistributed intoany other WANcontrol planeInfra Tenant L3OutConfiguration onLocal APICFirst-hop L3 devicereceives reachabilityinformation to centralAPIC nodesIP WAN IPNDHCP requests are2 relayed to the centralAPIC cluster by thefirst-hop L3 deviceOSPFOSPF4DHCP response3 sent to the RelayRL Node generates DHCP1 requests on uplinks(tagged with VLAN 4)AgentDHCP responsereceived by the RLNode allowing thenits full provisioningRL NodeRemote Leaf SiteACI Main DC19 - 21 March 2018 Cisco Connect Rovinj, Croatia31
ACI Remote Physical LeafEstablishing End-to-End IP ConnectivityFirst-hop L3 devicereceives reachabilityinformation to RLnodeOSPF can beredistributed into anyother WAN controlplaneIP WAN IPNOSPFOSPFRL nodeconfigurationfully provisionedOSPFRL TEPsassigned to thenodeRL NodeRemote Leaf SiteACI Main DC19 - 21 March 2018 Cisco Connect Rovinj, Croatia32
ACI Remote Physical LeafCOOP for Announcing Remote Endpoint InformationCOOP Sessionsbetween RLsnodes and spinesCOOP DBEP1RL TEPEP2RL TEPIP WAN IPNAnycastAnycastVTEPfor RLsVTEPfor (RLRLsTEP)(RL TEP)RL Local TablePo1EP1Po2EP1vPC Po1EP2vPC Po2EP2Remote Leaf SiteACI Main DC19 - 21 March 2018 Cisco Connect Rovinj, Croatia33
ACI Remote Physical LeafEP-to-EP Flow Local to RL Site (vPC)EP info syncover vPCcontrol planeIP WAN IPNRL1 Local TableEP1vPC Po1EP2vPC Po2Po1EP1‘Greedy Forwarding’vPC Po1 to Po2 on RLACI Main DC19 - 21 March 2018 Cisco Connect Rovinj, CroatiaPo2EP2Remote Leaf Site34
ACI Remote Physical LeafEP-to-EP Flow Local to RL Site (vPC Link Failure)IP WAN IPNRL1 Local TableEP1vPC Po1Po1EP2XPo2RL2 TEPEP1EP2Remote Leaf SiteACI Main DC19 - 21 March 2018 Cisco Connect Rovinj, Croatia35
ACI Remote Physical LeafCommunication between Endpoints in Separate RL SitesRL Site1RL Site 2IP WANTraffic betweenendpoints part ofseparate RL sitesalways flows throughthe spines CiscoDCConnectACI Main19 - 21 March 2018 Rovinj, Croatia36
ACI Remote Physical Leaf and Multi-PodRL Sites Can Be Associated to Separate PodsRL Site 2associatedto Pod2RL Site 1associatedto Pod1RL Site1IP WANInter-Pod Network(Multicast Enabled)IP WANACI Main DC Pod119 - 21 March 2018RL Site 2ACI Main DC Pod2 Cisco Connect Rovinj, Croatia37
ACI Remote Physical Leaf and Multi-PodRL Sites Can Be Associated to Separate Pods (Data Plane)RL Site 1IP WANInter-Pod Network(Multicast Enabled)IP WANACI Main DC Pod119 - 21 March 2018RL Site 2ACI Main DC Pod2 Cisco Connect Rovinj, Croatia38
ACI Remote Physical Leaf and Multi-SiteRL Sites Can Be Associated to Separate PodsRL Site 1associatedto ACI Site 1RL Site 2associatedto ACI Site 2Future ACIReleaseRL Site 1IP WANInter-Site NetworkIP WANACI Site 119 - 21 March 2018RL Site 2ACI Site 2 Cisco Connect Rovinj, Croatia39
ACI Remote Virtual Leaf (vPod)
Cisco ACI Virtual EdgeDecoupled From Hypervisor Kernel API DependenciesQ2FY18ACI 3.1ACI Virtual EdgeUserSpaceLegacy AVSVMVMCisco AVE (ACI 3.1 Release)VMVMAVEKernelSpaceACI Virtual Edge (AVE)VMVMVMVMVMVMAVSSwitching Policy EnforcementVMPolicy Enforcement,Services, TelemetryNative vSwitchVMHypervisor DependentMaintain ExistingOperational ModelsAVEVMSimple Transition/MigrationAVS AVE19 - 21 March 2018 Cisco ConnectHypervisor AgnosticPolicy Consistency AcrossMultiple Hypervisors Rovinj, CroatiaAVS/AVEFeature Parity41
Cisco ACI Virtual Pod (vPod)Extend ACI To Bare-Metal CloudIP NetworkLogical Connection To Spine(BGP-EVPN/ VXLAN)Virtual PodvSpinevLeafvLeafACI Virtual EdgeVMVMVMVMVMBare Metal CloudsVMVM(IBM BlueMix, AWS Elastic Metal etc.)RemoteLocationOn-PremiseColo FacilitiesRemote DataCenters19 - 21 March 2018 Cisco Connect(Equinix, CoreSite etc.) Rovinj, ts42
vPod Functional ComponentsvSpine, vLeaf, and AVEvPodvSpine vLeaf Run as container services inside VMs at the vPodlocation (co-located for availability)vLeaf: Distribute APIC policies to AVE forwarders(DME/PE)vSpine: Centralized endpoint and LPM database(COOP and BGP)Not in forwarding data pathAVE Implements ACI data path functionsUse iVXLAN for communication within Remote siteas well as between the vPod and other Pods19 - 21 March 2018 Cisco Connect Rovinj, CroatiavSpine vLeafDME/PECOOPOracleBGPRRIP NetworkiVXLAN ppDB43
vSpine, vLeaf, and AVEScale OutBGP EVPN Sessions between exClientAVEInfrastructure Services PECOOPCitizenBGPHypervisorHypervisor254 AVECiscoperConnectlogical Rovinj,vPodCroatia19 - 21 March pervisor44
vSpine, vLeaf, and AVEFlow State and E/PEDME/PEInfrastructure Services tate of Flow carried between AVE nodes during exClientAVECOOPCitizenBGPHypervisor19 - 21 March 2018COOPCitizenOPFlexClientBGPAVEHypervisor Cisco Connect Rovinj, exClientAVECOOPCitizenBGPHypervisor45
vPodInter-Pod Control PlaneDME/PEBGP EVPN Peeringbetween ‘spines’COOPOracleBGPRRiVXLAN hHypervisorHypervisorHypervisorWebAppiVXLAN OverlayAVEDBHypervisorWeb19 - 21 March 2018 Cisco Connect Rovinj, CroatiaAVEHypervisorAppAVEHypervisorDB46
vPodInter-Pod Data PlaneiVXLAN AN rHypervisorHypervisorWebAppiVXLAN OverlayAVEDBHypervisorWeb19 - 21 March 2018 Cisco Connect Rovinj, CroatiaAVEHypervisorAppAVEHypervisorDB47
ACI Extensions to Multi-Cloud
ACI Extensions To Multi-CloudACI Multi-SiteOrchestratorSite ASite C“Cloud APIC (cAPIC)”Site DSite BVMVMVMVMConsistent Network andPolicy across cloudsCommonGovernance19 - 21 March 2018 Cisco Connect Single Point ofOrchestrationRovinj, CroatiaVMVMSecure AutomatedConnectivity49
cAPIC and Policy MappingAWS Cloud ConstructsAPICUser AccountTenantVirtual Private NetworkVRFBD SubnetVPC subnetEPGSecurity GroupSecurity Group RuleEPG ContractsOutbound ruleConsumed contractsSource/Destination: Subnet or IP or Any or ‘Internet’ProtocolPortInbound ruleProvided contractsEC2 InstanceNetwork Adapter19 - 21 March 2018 Cisco Connect Rovinj, Croatia50
cAPIC and Policy MappingAzure Cloud ConstructsAPICTenantResource GroupVirtual NetworkVRFBD SubnetSubnetApplication Security Group(ASG)EPGNetwork Security Group(NSG)EPG ContractsOutbound ruleConsumed contractsSource/Destination: ASG or Subnet or IP or Any or ‘Internet’ProtocolPortInbound ruleProvided contractsVirtual MachineNetwork Adapter19 - 21 March 2018 Cisco Connect Rovinj, Croatia51
ACI AnywhereWhere to Go for More Informationü ACI Multi-Pod White ode refreshü ACI Multi-Pod Configuration icinfrastructure/white-paper-c11-739714.htmlü ACI Multi-Site White icinfrastructure/white-paper-c11-739609.htmlü ACI Multi-Pod Cisco Live Barcelona rary/?search weston#/session/BRKACI-2003ü ACI Multi-Site Cisco Live Barcelona rary/?search ardica#/session/BRKACI-2125ü ACI Physical Remote Leaf White PaperComing soon!19 - 21 March 2018 Cisco Connect Rovinj, Croatia52
Leaf to Leaf VTEP, Class -ID is local to the Fabric Leaf to Leaf VTEP, Class -ID is local to the Fabric . Support planned for a future ACI release for FX line cards and 9364C platform VXLAN Encrypted Fabric to Fabric Traffic [ GCM-AES-128 (32-bit PN), GCM--AES-256 . ACI Remote Physical Leaf Bu
The ACI Manual of Concrete Practice is a seven-part compilation of current ACI standards and committee reports. Part 1—ACI 117-10 to ACI 228.1R-03 Part 2—ACI 228.2R-13 to ACI 314R-11 Part 3—ACI 318-14 to ACI 346-09 Part 4—ACI 347R-14 to ACI 355.2-07 Part 5—ACI 355.3R-11 to ACI 440R-07 Part 6—ACI 440.1R-06 to ACI 533.1R-02
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
Chính Văn.- Còn đức Thế tôn thì tuệ giác cực kỳ trong sạch 8: hiện hành bất nhị 9, đạt đến vô tướng 10, đứng vào chỗ đứng của các đức Thế tôn 11, thể hiện tính bình đẳng của các Ngài, đến chỗ không còn chướng ngại 12, giáo pháp không thể khuynh đảo, tâm thức không bị cản trở, cái được
Cracking: ACI 224.1R, ACI 562, ACI 364.1R, and ACI RAP Bulletins Spalling/scaling: ACI 562, ACI 364.1R, ACI 506R, and ACI RAP Bulletins AMP should reference criteria used to determine which inspection results will require either: An Action Request
