Containerized Network Functions On Virtual Machines Or .
W H I T E PA P E R – N OV E M B E R 2 0 2 0Containerized NetworkFunctions on VirtualMachines or Bare Metal?Securing, Managing, and OptimizingCNFs and 5G Services at Scale
Containerized Network Functions on Virtual Machines or Bare Metal?Table of ContentsExecutive Summary3Introduction4Virtual Machines, Bare Metal, and the Transition to 5G5VMware Telco Cloud Platform6Cloud-native technology and cloud-first automation7Performance7Boosting performance by selecting a Linux kernel version8Performance in production environments8Security9NIST guidelines for securing containers9Containers alone are inadequate security boundaries9Risks of misconfiguration on a physical host10Securing the orchestration system10Taking advantage of advanced trends11Securing microservices with VMs11NCSC requirements for telecom security11Built-in security for virtual machines12European Union toolkit for cybersecurity of 5G networks12Infrastructure Management, IT Operations, and Lifecycle Management13Availability13Resource Management14Intent-based placement through service-aware infrastructure14Dynamic resource allocation and late binding for optimization14Data Persistence14Scalability15Networking15Container networking for Kubernetes clusters16Accelerating workloads and application-response times17Workload acceleration with SR-IOV17Automation17Conclusion: Management, Security, and Automation18W H I T E PA P E R 2
Containerized Network Functions on Virtual Machines or Bare Metal?Executive SummaryCSPs are turning to containers to streamline and scale the deployment of networkfunctions and 5G services. A container wraps a network function in a consistent, portablepackage that can be independently distributed and modified with little effort and fewdependencies. Containers then run on a host operating system and share its kernel. Thehost operating system resides on either a virtual machine or a physical server.Cost-effectively putting containerized network functions (CNFs) into production hinges onyour ability to secure, manage, and automate them at scale in an efficient and integralway. This paper explains how running containers on VMs establishes the perfect catalystfor efficiently and securely operating CNFs at scale. Combining containers and VMsproduces a powerful synergy that taps the benefits of both technologies.Virtual machines let you securely and efficiently run containerized functions and 5Gservices on software-defined infrastructure that you can easily manage, monitor, scale,automate, and optimize. Bare metal servers, in contrast, can root existing monolithicstacks in place and, in a multi-vendor environment, create silos, making management,automation, and maintenance difficult. Adding CNFs and an orchestrator like Kubernetesto a multi-vendor bare metal environment can compound complexity and furthercomplicate management.“Although“containers aresometimes thought of as thenext phase of virtualization,surpassing hardwarevirtualization, the reality formost organizations is lessabout revolution than evolution.Containers and hardwarevirtualization not only can, butvery frequently do, coexist welland actually enhance eachother’s capabilities. VMs providemany benefits, such as strongisolation, OS automation, anda wide and deep ecosystem ofsolutions. Organizations do notneed to make a choice betweencontainers and VMs. Instead,organizations can continue touse VMs to deploy, partition, andmanage their hardware, whileusing containers to package theirapps and utilize each VM moreefficiently.”APPLICATION CONTAINER SECURITY GUIDE, NISTSPECIAL PUBLICATION 800-190Hardware virtualization was originally developed to address the pain of working withphysical hardware, pain that ranges from time-consuming management problems andcash-consuming underutilization to the difficulty of scaling hardware for an elasticworkload. By optimizing utilization and simplifying management, virtualization reducesphysical hardware costs while improving scalability. The ease of scalability that comes withvirtualization is one of the reasons why major public cloud providers use hypervisors andVMs to run containers.For CSPs, performance, security, and management are key factors. Many of the studiesthat compare container performance on virtual machines with bare metal overlook theintegral requirements of securing and managing containers in a real-world environment. Studies show that optimizations in the vSphere CPU scheduler for NUMA architecturesquashes the belief that running containers on VMs comes with a performance tax. Noisy neighbor situations can cause interference for co-located containers onphysical hardware, and cross-container interference can result from containerssharing the same kernel resources or components. Kubernetes on bare metal is unlikely to outperform Kubernetes on VMware vSphere,which uses advanced scheduling algorithms to optimize all workloads. A recent test ofvSphere 7 with Kubernetes shows better performance compared with a bare-metalKubernetes node because the VMware hypervisor does a better job at schedulingpods on the right CPUs, thereby reducing random memory accesses. Containers alone are inadequate security boundaries; containers do not establishsecurity boundaries and strong isolation as VMs do. Running CNFs on bare metal would create a complex patchwork of bolted-on securitycontrols and tools. In contrast, running CNFs on virtual machines lets you imposesecurity by using built-in mechanisms that can be managed at scale without silos. Running containers on physical hardware would resurrect difficult infrastructuremanagement and operational problems that hardware virtualization solved years ago. Operating containers in production requires lifecycle management, high availability,resource management, data persistence, networking, and automation.Using VMware Telco Cloud Platform to run and automate containers on virtual machinesinstead bare metal satisfies the complete set of operational, management, and securityrequirements for deploying CNFs in production.W H I T E PA P E R 3
Containerized Network Functions on Virtual Machines or Bare Metal?IntroductionCommunications service providers are increasingly turning toward containers to acceleratethe development and deployment of network functions and 5G services.Containerization is a form a operating system virtualization. A container holds a selfdescribed application and the software components the application requires. Thecontainer runs on a container host operating system like Linux, which provides thecontainer with the components of an operating system, such as the kernel, hardwarescheduler, memory page abstraction, and the user space. With more than one container,the containers share the same underlying operating system. The container host in turnresides on either a virtual machine (VM) or a physical server (often referred to as baremetal).Because each container is self-describing, specifying the computing and networkingresources that it needs, it packages an application in a consistent, reproducible way: It canbe distributed, reused, and managed with minimal effort and few or no dependencies.Embodied in the term cloud-native technologies, this trend is advanced by using amicroservices architecture and a container orchestration system—typically Kubernetes.Microservices break up the functions of an application into a set of small, discreteprocesses, each of which can be independently developed, deployed, modified, andscaled. Kubernetes automates the deployment and management of containerizedapplications at scale.Running containerized network functions (CNFs) in production in a telecommunicationsnetwork comes with an established set of operational requirements: security, compliance,resource management, scalability, availability, data persistence, networking, andmonitoring. CNFs carry an additional requirement: orchestration.For CSPs, performance is another typical requirement, but although the performance ofcontainers on virtual machines and bare metal is comparable, putting containers intoproduction in a cost-effective and operationally efficient way hinges on your ability tosecure and manage containers at scale in an integral way.You can, at significant risk and expense, build a custom stack on physical hardware to tryto fulfill your containerized functions’ requirements, or you can use proven, cost-effective,low-risk virtualization solutions as the underlying infrastructure for managing, securing,and orchestrating containerized functions.But there is more: Combining containers and VMs taps the benefits of each technology,creating an organized whole that is greater than the sum of its parts—which is one reasonwhy the major cloud providers, such as Google and Amazon, use VMs to run containers.1Virtual machines let you securely and efficiently run containerized functions and 5Gservices in production on software-defined infrastructure that you can easily manage,monitor, scale, automate, and optimize. Containers, meanwhile, empower you to makedevelopers more agile, functions more portable, and deployments more automatable. Thecombination of the two streamlines the development, deployment, and management ofCNFs.This paper explains how running containers on VMs establishes the perfect catalyst forreliably and robustly operating containerized functions at scale. VMware Telco CloudPlatform , which uses Kubernetes to orchestrate containers on virtual machines in asoftware-defined data center and a telco cloud, stands at the center of this combination.1 Combining containers and virtual machines to enhance isolation and extend functionality on cloud computing, Ilias Mavridis,Helen Karatza, Future Generation Computer Systems, Volume 94, 2019, Pages 674-696, ISSN 0167-739X, https://doi.org/10.1016/j.future.2018.12.035.W H I T E PA P E R 4
Containerized Network Functions on Virtual Machines or Bare Metal?“Containers“promise bare metalperformance, but as we haveshown, they may suffer fromperformance interferencein multi-tenant scenarios.Containers share the underlyingOS kernel, and this contributesto the lack of isolation. UnlikeVMs, which have strict resourcelimits, the containers also allowsoft limits, which are helpfulin overcommitment scenarios,since they may use underutilizedresources allocated to othercontainers. The lack of isolationand more efficient resourcesharing due to soft-limits makesrunning containers inside VMs aviable architecture.”CONTAINERS AND VIRTUAL MACHINES AT SCALE:A COMPARATIVE STUDYVirtual Machines, Bare Metal, and the Transition to 5GAmid a backdrop of fierce competition and digital transformation, communications serviceproviders seek to develop new business models, simplify operations, and launch newservices, all in a quest to increase revenue and expand profit margins. Although 5G opensup new business opportunities, the complex, siloed architecture of CSPs’ existingnetworks stands in the way of rapid innovation and operational agility, hampering thedigital transformation.These existing networks, which tend to be founded on vertically integrated monolithicstacks designed to run vendor-specific virtual network functions (VNFs), make automatingdeployment and management difficult. Bare metal servers root these monolithic stacks inplace and can, especially in a multi-vendor environment, create difficult-to-manage silos.In this environment, maintenance updates can spiral into a complex cycle.If one of the silos needs an update, for example, you must also check whether thehardware is still supported. Likewise: Have the north-bound APIs of the managementsystem like VNFM changed? Are the VNMFs using any old APIs, or will the VNFM nowneed to be updated? If the VNFM is updated, will the VNF also need updating? Is there anautomation layer using the old VNFM APIs, or will the automation layer also need to beupgraded? If there are hardware differences among the servers, additional components,such as drivers, will likely also need attention. The more silos there are, the greater thechallenge.When CSPs turn to cloud-native technology to run network functions in containers onbare metal alongside VNFs in multi-vendor environments, the complexity spirals furtherout of control. CNFs require additional interfaces and tools beyond those used by VNFs—such as Kubernetes clusters, container networking interfaces, container image registries,minimalist Linux container hosts, and tools like Helm and Docker—that would make thestack even more difficult to visualize, secure, operate, and maintain.In this way, infrastructure that relies too heavily on physical hardware without exploitingthe abstraction that virtual machines provide makes it difficult to automate multi-tenant,distributed containerized network functions and to deliver the resiliency and reliabilitythat’s required in a highly regulated industry with strict service-level agreements anddemanding consumers. Several emerging telecommunications regulations, for example,promote security and resiliency through supplier diversity.To achieve web-scale speed and agility while maintaining carrier-grade performance andquality, CSPs need a platform that combines telco-specific cloud-native solutions andcloud-first automation with consistent infrastructure. CSPs must be able to automate andorchestrate their functions and services across systems from multiple vendors.The following elements are critical to establishing a modern holistic multi-vendor platformwith the power to innovate quickly, scale with elasticity, adopt a multi-cloud strategy, andmanage functions and services efficiently: Hybrid infrastructure that spans multiple clouds and sites, from the core and the edgeto private and public clouds, so you can run hybrid network services that combinefunctions in different formats. Cloud-native technology such as containers and Kubernetes that lets you build,manage, and run containerized network functions (CNFs) across distributed sites. Multi-layer, cloud-first automation that unites your infrastructure and multi-cloudresources, including containers and VMs, in a centralized orchestration system.W H I T E PA P E R 5
Containerized Network Functions on Virtual Machines or Bare Metal?Figure 1: VMware Telco Cloud Automation and VMware Telco Cloud Infrastructure work together to run and manage CNFsand containerized 5G applications on consistent horizontal infrastructure.VMware Telco Cloud PlatformThis paper explains how VMware Telco Cloud Platform and its components radicallysimplify security, operations, and management of 5G networks and functions by runningCNFs on virtual machines. A quick overview of VMware Telco Cloud Platform helps solidifythe concepts that appear later in the paper.By solving the problems that undermine the architecture of existing telecommunicationsnetworks—monolithic stacks marred by complexity, silos, vendor lock-in, and an overreliance on old physical hardware—VMware Telco Cloud Platform empowers CSPs toreduce operational complexity and launch innovative services on consistent infrastructure.The two fundamental elements of VMware Telco Cloud Platform are VMware Telco CloudInfrastructure and VMware Telco Cloud Automation .VMware Telco Cloud Automation orchestrates network functions, services, and resourcesfrom a centralized location. The platform integrates with multiple virtual infrastructuremanagers (VIMs) and Kubernetes clusters to form a powerful multi-tenant environment tosecurely manage the service and application layer. The platform uses VMware TanzuStandard for Telco to orchestrate containers, and VMware Telco Cloud Automationcentralizes the provisioning and management of the Kubernetes clusters.VMware Telco Cloud Infrastructure supplies infrastructure as a service (IaaS) andcontainers as a service (CaaS) with the following virtualization technology: VMwarevSphere, VMware NSX-T Data Center, and VMware vSAN . The deployment andmanagement of virtual machines on vSphere furnishes the foundation for running CNFsand VNFs. Tanzu Standard for Telco provides a carrier-grade Kubernetes distribution withtelco-grade extensions to run and manage CNFs at scale.VMware Telco Cloud Platform can be deployed across 5G networks to meet target designand scalability objectives. The VMware telco cloud reference architecture simplifies theplatform’s implementation and describes how the platform gives you a flexible foundationto fulfill various 5G use cases.By enabling you to deploy virtual network functions (VNFs) and containerized networkfunctions (CNFs) on a consistent horizontal infrastructure, your CSP can evolve frominfrastructure as a service to containers as a service.W H I T E PA P E R 6
Containerized Network Functions on Virtual Machines or Bare Metal?“Applications“can benefit fromthe security and performanceisolation provided by the VM,and still take advantage of theprovisioning and deploymentaspects of containers. Thisapproach is quite popular, and isused to run containers in publicclouds where isolation andsecurity are important concerns.”CONTAINERS AND VIRTUAL MACHINES AT SCALE:A COMPARATIVE STUDYCloud-native technology and cloud-first automationCapitalizing on the opportunities of 5G in a multi-cloud world hinges on two keyingredients of VMware Telco Cloud Platform: CaaS and cloud-first automation.Containers and Kubernetes decouple network functions from the infrastructure so theycan be deployed quickly, shared among services, updated easily, and managedindependently. Orchestration and automation dynamically scale network functions tomeet changes in demand. By implementing containers as a service (CaaS), CSPs can usethe same technology to meet different requirements across their 5G networks. As a result,CSPs can design more efficient 5G networks.Cloud-first automation unites multi-cloud resources in a centralized orchestration systemand then uses intent-based placement for optimization. With cloud-first automation, whichcontinuously synchronizes with registered clouds, CSPs obtain context-aware informationabout their diverse set of sites, the state of these sites, the applications running there, theembedded technologies available to foster service delivery, and the cloud resourcesavailable for allocation. When the orchestrator can access this information, it canrecommend placement of network services and functions in a way that alignsrequirements with available cloud resources and capabilities. In this way, cloud-firstautomation further simplifies and optimizes the deployment and management of CNFs.PerformanceThe CPU scheduler of VMware ESXi enables the hypervisor to provide equivalent orbetter overall workload performance for containers than multi-purpose Linux operatingsystems running on physical hardware.A comparative study by VMware shows that an enterprise web application can run inDocker containers on vSphere 6.5 with better performance than Docker containers onbare metal, largely because of optimizations in the vSphere CPU scheduler for nonuniformmemory access (NUMA) architectures, quashing the belief that running containers on VMscomes with a performance tax.2 vSphere is better at scheduling VMs on NUMA nodeswhere their memory resides. Linux, on the other hand, tries to maximize processorutilization, meaning processes may be scheduled on different NUMA nodes from theirmemory, slowing memory access and degrading performance. A performance analysis ofbig data workloads on vSphere shows the same results.Virtualization can offer better performance isolation than running containers in Linux,especially in noisy neighbor situations. The results of an academic comparative study ofcontainers and VMs at scale show that “co-located applications can cause performanceinterference, and the degree of interference is higher in the case of containers for certaintypes of workloads.”3Because of how the Linux kernel works, you can also get cross-container interferencefrom containers sharing the same kern
microservices architecture and a container orchestration system—typically Kubernetes. Microservices break up the functions of an application into a set of small, discrete processes, each of which can be independently developed, deployed, modified, and scaled. Kubernetes automates the deployment and management of containerized applications at .
FOR NETWORK FUNCTIONS VIRTUALIZATION NETWORK FUNCTIONS VIRTUALIZATION: A PRIMER 3 VIRTUALIZING NETWORK FUNCTIONS: COULD NFV MEAN NETWORK NIRVANA? NEW MODELS New Management Models Needed for NFV When a cadre of giant global network oper-ators started the initiative known as Network Functions Virtualization (NFV) in late 2012,
Viewing the Virtual Network virsh net-list --all Deleting the Default Network virsh net-undefine default Creating Virtual Network virsh net-define xml_file_name Viewing the Virtual Network virsh net-list --all Starting the Virtual Network virsh net-start network_name_that is in the list Example: [root@localhost ]# virsh net-list --all
Each NETLAB remote PC or remote server runs inside of a virtual machine. VMware ESXi provides virtual CPU, virtual memory, virtual disk drives, virtual networking interface cards, and other virtual hardware for each virtual machine. ESXi also provides the concept of a virtual networking switch.
"Virtual PC Integration Components" software must be installed into each virtual machine. In a Windows host, the "Virtual PC Integration Components" software for a Windows virtual machine is located at C:\Program Files (x86)\Windows Virtual PC\Integration Components\ Multiple virtual machines can access the same target folder on the host.
network.edgecount Return the Number of Edges in a Network Object network.edgelabel Plots a label corresponding to an edge in a network plot. network.extraction Extraction and Replacement Operators for Network Objects network.indicators Indicator Functions for Network Properties network.initialize Initialize a Network Class Object
Virtualization as a Paradigm Virtual Machine Guest OS Guest Applic ation Virtual Machine Guest OS Guest Applic ation Virtual Machine Guest OS Guest Applic ation NF: Network Function VNF: Virtual Network Function NC: Network Controller VN: Virtual Network Virtualiz ation and Applicati on Manage men Cloud Open Hardware Host OS HypervisorNaaS .
Wide area virtual network Data Cente r 1 Data Cente r 2 Data Cente r 3 L2 over L3 tunnel Virtual Network A (Low-latency) Virtual Network B (Bandwidth reserved)networks on an existing L3 network by Virtual Network C (Low-cost, best effort) Tenant A Tenant B Tenant C Tenant A Tenant B Tenant C Data Center 21 Data Center 3 Benefit
tank; 2. Oil composition and API gravity; 3. Tank operating characteristics (e.g., sales flow rates, size of tank); and 4. Ambient temperatures. There are two approaches to estimating the quantity of vapor emissions from crude oil tanks. Both use the gas-oil ratio (GOR) at a given pressure and temperature and are expressed in standard cubic feet per barrel of oil (scf per bbl). This process is .
