Running Legacy VM’s Along With Containers In Kubernetes
Running Legacy VM’s along with containers in KubernetesDelusion or Reality?Kunal KushwahaNTT Open Source Software CenterCopyright 2019 NTT Corp. All Rights Reserved.
About me Work @ NTT Open Source Software CenterCollaborator (Core developer) for libpod (podman)Contributor KubeVirt, buildkit and other related projectsDocker Community Leader @ Tokyo ChapterCopyright 2019 NTT Corp. All Rights Reserved.2
Growth of Containers in CompaniesAdoption of containers in production has significantly increasedCredits: CNCF websiteCopyright 2019 NTT Corp. All Rights Reserved.3
Growth of Container Orchestration usageAdoption of container orchestrator like Kubernetes have also increasedsignificantly on public as well private clouds.Credits: CNCF websiteCopyright 2019 NTT Corp. All Rights Reserved.4
Infrastructure p-1app-NVMkernelVM PlatformVM PlatformExisting ProductsNew Products The application infrastructure is fragmented as most of old application stillrunning on traditional infrastructure. Fragmentation means more work & increase in costCopyright 2019 NTT Corp. All Rights Reserved.5
What keeps applications away from Containers Lack of knowledge / Too complex to migrate in containers.Dependency on custom kernel parameters.Application designed for a custom kernel.Application towards the end of life.Companies prefer to re-write application, rather than directly migrating them rchestration-and-beyondCopyright 2019 NTT Corp. All Rights Reserved.6
Ideal pp-NVMkernelVM Platform Applications in VM and containers can be managed with same control plane Management/ Governance Policies like RBAC, Network etc. can same for all application Intercommunication between application over containers and VM possible.Copyright 2019 NTT Corp. All Rights Reserved.7
“Lift & Shift” Strategy of Migration Original terminology coined for migrating inhouse application to Cloud. Also known as re-hosting application. The lift and shift migration approach is about migrating your application andassociated data to the target platform with minimal or no changes. Making VMs part of Kubernetes infrastructure along with containers, will help Lift &Shift strategy for migrating applications running in VMs to Kubernetes.*Pic credit: Google CloudCopyright 2019 NTT Corp. All Rights Reserved.8
KubeVirt Overview KubeVirt extends Kubernetes by adding resource types for VMs throughKubernetes Custom Resource Definitions API It enables to run VMs along with containers on existing Kubernetes nodes VMs run inside regular Kubernetes pods, where they have access tostandard pod networking and storage, and managed using standardKubernetes tools such as kubectl Build on mature technology like KVM, qemu, libvirtd, KubernetesCopyright 2019 NTT Corp. All Rights Reserved.9
KubeVirt GoalsLeverage KubeVirt andKubernetes to managevirtual machines forimpractical-to-containerizeapps.Combine existingvirtualized workloads withnew container workloadson the one platform.Support development ofnew micro-serviceapplications in containersthat interact with existingvirtualized applications.Copyright 2019 NTT Corp. All Rights Reserved.10
KubeVirt Control Flow & Architecture Virt-API and Virt-Controller are added to Master Node.Virt-Handler is on each worker node, responsible to launch VM in a pod.Containerized-data-importer prepare persistent VolumesCopyright 2019 NTT Corp. All Rights Reserved.11
Important Features of KubeVirt KubeVirt features Can be installed and removed in existing k8s cluster.Supports multiple network and storage options, suitable for migrationVMs run as part of pod, so utilize all other k8s components like DNS, RBAC, NetworkPolicies etc.VM capabilities Run VM with images in qemu qcow2 format, same as in OpenStacklatest device support Q35 machine support.Copyright 2019 NTT Corp. All Rights Reserved.12
KubeVirt Evaluation ProcessCopyright 2019 NTT Corp. All Rights Reserved.13
Evaluation ViewpointVM to K8s Image migration Import into k8s PV or Container Image Understand problems/limitations of systemConfiguration & Deployment Design VM to match original requirements / environment Understand problems/limitations /workaroundsOperational & Functional Validation Service creation App functionality/ accessibility / restrictionReliability Time to recover from failure Maintenance downtime/disruptionCopyright 2019 NTT Corp. All Rights Reserved.14
Important KubeVirt Objects VirtualMachine (VM) :represents a virtual machine in the runtime environment of Kubernetes. VirtualMachineInstanceReplicaSet (VMRS) :Tries to ensures that a specified number of virtual machine replicas are running at any time. DataVolume :Data Volumes(DV) are an abstraction on top of Persistent Volume Claims(PVC) and theContainerized Data Importer(CDI) ContainerRegistryDisk:local ephemeral disk for booting VMI. Any changes to disk are not persisted across reboot.Copyright 2019 NTT Corp. All Rights Reserved.15
data stateconvert inqcow2 format Import inPersistentfile for VM DeployapplicationDefinition inwith kubectlKubeVirtapplyExpose Service ConsistentDisk & Prepare yamlDeployfor Migration Export the VMPrepare k8s Manifest Prepare VMExport & Build ImageApp in VMMigration of VM to KubeVirt CreateService Expose theservice toouter networkVolume (PV)Or Build DockerimageCopyright 2019 NTT Corp. All Rights Reserved.16
Measuring ParametersMigration effort Image Migration Configuration & DeploymentConfiguration&DeploymentReliability Maintenance Reliability of serviceMaintenanceHardMediumEasyCopyright 2019 NTT Corp. All Rights Reserved.17
Use Casesreplica: 1Copyright 2019 NTT Corp. All Rights Reserved.18
Evaluation Use Case Monolithic Application (Single VM) 3 Tier Web Application (Multiple VM) HA with multi network ArchitectureCopyright 2019 NTT Corp. All Rights Reserved.19
Monolithic Applicationreplica: 1Copyright 2019 NTT Corp. All Rights Reserved.20
Monolithic ICMonolithic Appin VMVM Platform (oVirt / ESXi .)Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.21
Monolithic IC Monolithic Appin VMPersistentdataApplication stores the data in filebased DB locally of diskVM Platform (oVirt / ESXi .)Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.22
Monolithic ation TypeMonolithic AppRunningin VM DB.Standalone application withfile basedUsersRequirements Persistent StorageNetworkingVolume BackupVM Platform (oVirt / ESXi .)Policies No auto re-creation of VMHealth CheckCompany NetworkCopyright 2019 NTT Corp. All Rights Reserved.23
Migration process: Image MigrationImage Migration is simple process- Depending on disk size, it may be time consuming. Converting vm-disk to kubevirt compatible format img, qcow2, iso etc are supported formats *Conversion can be done with any v2v or p2v tools Importing disk to KubeVirt (Kubernetes) qemu-img convert –f vdi monolithic.vdi –O qcow2 mono.qcow2 virtctl image-upload –pvc-name-monolithic-vm-disk \--pvc-size 64Gi\--image-path /home/kunal/images/mono.qcow2 \--uploadproxy-url ainerized-data-importer/blob/master/doc/supported operations.mdCopyright 2019 NTT Corp. All Rights Reserved.24
Migration process: VM definition Depending on original VM configuration,writing VM yaml file could be tough.[1] Translation of old VM configuration tonew VM yaml is done manually. Key definitions run strategy : defines vm state after objectcreation (running, manual etc)VolumeNetworkapiVersion: kubevirt.io/v1alpha3kind: VirtualMachinemetadata:labels:kubevirt.io/vm: monolithic-appname: monolithic-appspec:runStrategy: manualtemplate:spec:terminationGracePeriodSeconds: 30domain:devices:disks:- disk:bus: virtioname: pvcdiskvolumes:- name: pvcdiskpersistentVolumeClaim:claimName: monolithic-vm-disknetworks:- name: defaultpod: {}[1] : OpenShift supports KubeVirt templates, which is helpfulCopyright 2019 NTT Corp. All Rights Reserved.25
Migration process: Service DefinitionCommon to Kubernetes All solutions of Service Discovery of Kubernetes shall work with KubeVirtVMs too.apiVersion: v1kind: Servicemetadata:name: intranet-servicesspec:ports:- name: nodeportnodePort: 30000port: 27017targetPort: 80selector:kubevirt.io/vm: monolithic-apptype: NodePortSample service definitionCopyright 2019 NTT Corp. All Rights Reserved.26
After Migration: Monolithic applicationDNSWorker NodeMonolithic eletCompany NetworkCopyright 2019 NTT Corp. All Rights Reserved.27
Migration process: MaintenanceKubernetes/KubeVirt do not add much value for maintenance phase for thiskind of application Backup/snapshot management. PersistentVolume (PV) is provided by K8s storage providers. Managed in similar way as PersistentVolume of K8s. Patch management/VM upgrade Traditional way (ssh / config manager) On failure Depending on Run strategy, action can be defined.Copyright 2019 NTT Corp. All Rights Reserved.28
Conclusion: Monolithic application migration Easy to migrate and maintain application in Kubernetes-Migration process :online migration :Security :-RBAC policiesNetwork policiesMaintenance:Reliability with Kubernetes on learnt-Migration effortAs good as M maintenance changes w.r.t. Kubernetes.Be expert in Kubernetes.Copyright 2019 NTT Corp. All Rights Reserved.29
3-Tire Web ApplicationCopyright 2019 NTT Corp. All Rights Reserved.30
3 Tier Web endhttp://webservices-intranet.comVM Platform (oVirt / ESXi .)Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.31
3 Tier Web ApplicationDNS Frontend & Application logic do notstore data locally.Backend store all data of applicationExternal network connect only frontendNo taVM Platform (oVirt / ESXi .)Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.32
3 Tier Web esBackendRequirementsApplicationLogicUsers3 tier web architecture.FrontendApplication Type Application and Frontend should be scalable.DBPersistent Storagefor Backendeth0NetworkingBackend Inter-VM & external communicationVolume Backup VM Platform (oVirt / ESXi .)Auto re-create ofApplication & Frontend VMNo auto re-creation of VM for BackendHealth Check Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.33
Migration process: Image Migration ContainerDisk type suites better for immutable application types. Extra temporary storage can be provided using EmptyDisk type.PersistentVolume(PV) for storing persistent data in application.Frontend and Application VM imported asContainerDiskContainerDisk is created using Dockerfilewith special Base Image provided byKubeVirt. cat DockerfileFROM kubevirt/container-disk-v1alphaADD frontend-disk.qcow2 /disk docker build –t kunalkushwaha/frontend-disk:v1Copyright 2019 NTT Corp. All Rights Reserved.34
Migration process: VM DefinitionFrontend and Application logic are created as VMReplicaSet To make Frontend and Application scalable,defined as VMReplicaSet(VMRS). Though VMs created using ContainerDisk arenot compatible with live-migration. Data/Configuration can be passed to applicationin VM using cloudInit or ConfigMap during VMcreation.spec:replica: 1devices:disks:- disk:bus: virtioname: containerdisk- disk:bus: virtioname: configdiskvolumes:- name: containerdiskcontainerDisk:image: kunalkushwaha/frontend-vm-disk:v1- name: configdiskcloudInitNoCloud:userDataBase64: (cat app-scripts.sh base64 -w0).Sample VMReplicaSet definitionCopyright 2019 NTT Corp. All Rights Reserved.35
Migration process: Service Definition Hostname of old topology system becomes service name Frontend exposed with NodePort Application and Backend as ClusterIP (accessed within Cluster)apiVersion: v1kind: Servicemetadata:name: web-serverspec:ports:- name: nodeportnodePort: 30002port: 27019targetPort: 80selector:kubevirt.io/vm: frontend-apptype: NodePortFrontendapiVersion: v1kind: Servicemetadata:name: application-serverspec:ports:- name: clusteripport: 27021targetPort: 80selector:kubevirt.io/vm: application-apptype: ClusterIPApplication &BackendCopyright 2019 NTT Corp. All Rights Reserved.36
After Migration: 3 Tier Web ApplicationDNSNodePortWorker ices-intranet.comUsersKubeletCompany NetworkCopyright 2019 NTT Corp. All Rights Reserved.37
Migration process: MaintenanceVMReplicaSet are easy to scale, same as Pod replicaset, But norolling updates supported. Blue-Green deployment for updating immutable VMs outside of KubeVirt. Scale with updated image.Delete old image instancesScale downUse traditional approach for updating Stateful VM instances. ssh, config managementCopyright 2019 NTT Corp. All Rights Reserved.38
Conclusion: 3 Tier Web Application Maintaining & scaling stateless VMs becomes very easy.-Migration process :Online migration :Maintenance :Reliability with Kubernetes :MediumNoGoodGoodMigration effortConfiguration&DeploymentReliabilityLesson learnt-Name resolution/ Fixed IP reference in applicationconfig, do not work.Hostname of VMs will be services of VM instance.Be expert in Kubernetes.MaintenanceHardMediumEasyCopyright 2019 NTT Corp. All Rights Reserved.39
HA ArchitectureCopyright 2019 NTT Corp. All Rights Reserved.40
HA Architecture Patterns Active-Standby with Shared DiskMasterStandby Active-Standby with Shared nothingMasterStandby Active-Active with Shared nothing**Please see appendicesMasterMasterCopyright 2019 NTT Corp. All Rights Reserved.41
HA Architecture ersNICVIPMasterVM Platform (oVirt / ESXi .)Company NetworkCopyright 2019 NTT Corp. All Rights Reserved.42
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)MasterStandbyCopyright 2019 NTT Corp. All Rights Reserved.43
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)Hellothere!MasterStandbyCopyright 2019 NTT Corp. All Rights Reserved.44
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)Hellothere!No ResponseLet me take overMasterStandbyCopyright 2019 NTT Corp. All Rights Reserved.45
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)Hellothere!MasterNo ResponseWhat if “He is stillon disk?”Letwritingme takeoverStandbyCopyright 2019 NTT Corp. All Rights Reserved.46
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)Hellothere!MasterPowerOffNo ResponseWhat if “He is stillon disk?”Letwritingme takeoverStandbyCopyright 2019 NTT Corp. All Rights Reserved.47
Active-Standby with Shared Disk Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.When one node become unresponsive. How it can be ensured if it is notupdating disk / Corrupting data? Shoot The Other Node In The Head (STONITH)Hellothere!No ResponseWhat if “He is stillon disk?”applicationsLetwritingme takeoverLack of fencing mechanism, restrict migration ofPowerOffimplementedwithSTONITH likesolutionMasterStandbyCopyright 2019 NTT Corp. All Rights Reserved.48
Active-Standby without Shared DiskVIPRead-WriteService - LANpostgresqlRead OnlypostgresqlDATA - LANPacemakerPacemakerInterConnect - LANcorosyncMastercorosyncStandbyCopyright 2019 NTT Corp. All Rights Reserved.49
Active-Standby without Shared DiskVIP- LAN PG-REX is a solution basedServiceon PostgreSQL& Pacemaker. Based on streaming replication feature. Open Sourcepostgresqltool for easier setup*postgresqlDATA - LANPacemakerPacemakerInterConnect - projects/pg-rex/Copyright 2019 NTT Corp. All Rights Reserved.50
Migration process: VM Definition of HA models Multus ( a meta CNI plugin) used for providing multiple networkinterfaces to VMs of KubeVirt. Uses NetworkAttachment (CNI CRD) for implementing multiplenetworks. Apart from Persistent Volume, this use case requires multiple Network segments. Preparation of network is required before using them in VM Definition i.e. definingNetworkAttchmentDefinition.Copyright 2019 NTT Corp. All Rights Reserved.51
Migration process: VM Definition of HA models Multus ( a meta CNI plugin) used for providing multiple networkinterfaces to VMs of KubeVirt. Uses NetworkAttachment (CNI CRD) for implementing multiplenetworks.apiVersion: v1kind: NetworkAttachmentDefinitionmetadata:name: pgrex-s-lanspec:config:’{“cniVersion”: “0.3.0”,“type”: “macvlan”,“master”:”enp0s8”,. CNI plugin .}’pgrex-s-lan.yaml Apart from Persistent Volume, this use case requires multiple Network segments. Preparation of network is required before using them in VM Definition i.e. definingNetworkAttchmentDefinition.Copyright 2019 NTT Corp. All Rights Reserved.52
Migration process: VM Definition for PG-REX With private hosted Kubernetes, its hard to get Fixed IP over cluster. Service cannot have custom ClusterIP in different segment.Migration in KubeVirt is possible with hackish solution.Works on fixed IP address, but troubleshooting is hard.Using macvlan network, network with narrow range of IP is be created for allsegments.HA components communicate with VM IP’s instead of services.Extra logic required to ensure user request goes to Master VM only.*Need reconfiguration, if VM’s moved from current node.*Leader election to mark Active VM. election-with-kubernetes/)Copyright 2019 NTT Corp. All Rights Reserved.53
After Migration: Active-Standby without Shared DiskDNSWorker ces.comKubeletUsersFixed IPCopyright 2019 NTT Corp. All Rights Reserved.54
Migration process: MaintenanceMaintenance approach of
Kubernetes Custom Resource Definitions API It enables to run VMs along with containers on existing Kubernetes nodes VMs run inside regular Kubernetes pods, where they have access to standard pod networking and storage, and managed using standard Kubernetes tools such as kubectl
2.16 Schematic of a legacy logo endorsement 2.17 Example of a legacy program logo endorsement 2.18 Example of a legacy agency logo endorsement 2.19 Legacy logo endorsement of a sub-program or initiative lock up 2.20 Schematic of a legacy logo endorsing a sub-program 2.21 Example of a legacy logo
network governance phases: legacy conceptualization, legacy planning and implementation, legacy transfer, and post-Games legacy governance, as well as a number of governance mechanisms (e.g., contracts, policies) that had an impact on the overall governance of the event's legacy. Finally, a critical analysis of the governance of
Legacy 2.5i Limited in Twilight Blue Metallic with optional equipment. Introducing the 2016 Subaru Legacy See ho far trust can take you. Confidence and trust are their own kind of freedom in the new 2016 Legacy, the longest-lasting midsize sedan in its class.1 Considering that 95% of all Subaru Legacy vehicles sold
Civil Liberties and the Legacy of Harry S. Truman The Truman Legacy Series, Volume 9 Based on the Ninth Truman Legacy Symposium The Civil Liberties Legacy of Harry S. Truman May 2011 . President Truman’s special message to Congress on Civil Rights, February 2, 1948 . . . 178
Dolomite Legacy Deluxe Rollator Padded backrest, padded seat and basket standard. The Dolomite Legacy is more than just a name, it is a Legacy. For over 10 years the Legacy has brought daily mobility back to hundreds of thousands throughout 80 countries worldwide. Founded on Dolomite’s patented flip-up seat technology, Legacy provides provenFile Size: 939KB
Dolomite Legacy Vægt Legacy 450 Legacy 520 Legacy 600 Vægt i alt1 6,9 kg/15,2 lbs 7.2 kg/15.9 lbs 7,6 kg/16 lbs Maks. brugervægt2 150 kg/330 lbs 150 kg/330 lbs 150 kg/330 Ibs Maks. belastningikurv 7 kg/15 lbs 9 kg/20 lbs 9 kg/20 lbs Maks. belastningpåbord 5 kg/11 lbs 5 kg/11 lbs
8 Dragon PTN Legacy Services Release 01 05/2020 1. INTRODUCTION 1.1 General This document is valid as of Dragon PTN Release 4.3DR. This manual describes the OAM for so-called legacy services that can be used to transport legacy applications over the Dragon PTN network. Legacy services
memory giving sector. Since then over 80 charities have taken part Legacy Monitor reaches 50 members! 2014 Legacy Monitor Netherlands is set up in partnership with local expert Arjen van Ketel and eight pioneer charities (now 19 and counting!). Legacy Giving 2050: Legacy Giving in an Age of Uncertainty assesses the long-term outlook
