Container-based Framework To Migrate Disparate IBM Integration Patterns .
WhitepaperContainer-based Framework to MigrateDisparate IBM Integration Patternson Any CloudBy Mithun Roy Thadi, Jennifer Gnanaraj, Naganjali Vinnakota, Kiran Kumar Jujjuri
Table of ContentAbstract. 3Introduction. 4Containers. 5IBM Integration Tech Stack on Cloud. 5Integration Framework for Cloudification. 6Integration Patterns:.12Transactions from MQ to MQ:.12Transactions from MQ to REST:.13Transactions from REST to MQ endpoint:.14Transactions from REST to REST endpoint:.15IBM APP Connect.17IBM API Connect.18IBM DB2.19IBM MQ.20Continuous Integration.21Conclusion.21Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 2
AbstractIn the era of cloud technologies and digital transformations, application modernization isa fundamental part for cloud-centric business transformation, which is inevitable. Digitaltransformation is a critical process for any company or organization regardless of size orindustry. To undergo a smooth and seamless digital transformation, a robust framework isrequired. These frameworks provide businesses with a roadmap of transformation strategiesto help companies survive disruption by adapting to current technologies.The scope of this document is completely limited to the IBM Integration technologies, preciselycovering a framework built on IBM App Connect. As cloud technology has advanced and tobecome more cloud-centric, IBM has made many changes to its IBM Integration technologieslike IBM MQ and IBM Integration Bus. It has provided IBM Cloud Pak for integration on IBMCloud and with RedHat OpenShift to run this Cloud Pak on other third-party clouds withoutcompromising any of its features.As everything is now available for integration technologies to run on any cloud, a digitalintegration transformation framework is required for companies to transform their currentbusinesses to this new environment without major changes. The framework needs to beable to be containerized, adaptable to any combination of integration patterns, and providea cloud experience everywhere.Since they are lightweight and portable, containers are ideal for application modernizationand hybrid, multi-cloud scenarios.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 3
IntroductionIn this modern era, the future of any technology highly depends upon its adaptability totechnological advancements. Adapting to cloud architecture is one such challenge fromwhich IBM Integration tech stack is not exempt i.e., IBM Integration Bus, IBM Message Queue,etc. IBM has addressed this concern by creating its own cloud and RedHat OpenShift forother clouds, which hosts multiple IBM technologies like IBM Cloud Pak for Integration (CP4I)on any cloud.IBM Cloud Pak for Integration (CP4I) is a hybrid integration platform with an automated,closed-loop approach that supports multiple styles of integration within a single, unifiedexperience.The following integration framework is designed to make IBM Integration tech stacka reusable generic solution that could be customized specifically to domain and clientrequirements, containerized on all clouds.This integration framework is entirely built on IBM App Connect and IBM MQ and can beused on any (Hybrid/Multi) cloud model.Figure 1: Integration FrameworkWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 4
ContainersContainers are executable units of software in ndencies are also included in common ways so thatit can be run anywhere: from the desktop, traditional IT, orthe cloud.Unlike a virtual machine, containers do not need to includea guest OS in every instance. They can, instead, simplyleverage the features and resources of the host OS. Thismakes them small, fast, and portable.Of late, we’ve been talking about applicationmodernization as one of the prominent benefits ofcontainers.IBM Integration TechStack on CloudThere are many cloud-providers like Amazon Web Services(AWS), Microsoft Azure, Google Cloud, IBM Cloud, OracleCloud, Alibaba Cloud, etc.IBM CloudOther CloudsDeployableIssue withother clouds?IBM has its separate tech stack (IBM Cloud Pak forIntegration (CP4I)) which is provided as SaaS on IBMCloud and on IBM Red Hat OpenShift.IBM Tech Stack (CP4I)IBM Red Hat OpenShift provides a consistent applicationplatform for the management of existing, modernized,and cloud-native applications (CP4I) that run on any cloud.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud Figure 2: IBM Tech Stack5
Integration Frameworkfor CloudificationIntegration transformation frameworks are essential for companies to transition to the cloudsmoothly and seamlessly.IBM ACECachePersistentVolumeGateway FlowCache RefresherFlowFile SystemDatabaseOptional StorageProcessor FlowsReplayDataPowerIBM MQMonitoringDispatcher FlowAudit and tion FrameworkFigure 3: Integration Transformation FrameworkWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 6
By categorizing the flows according to the functions they perform, the following three-flowdesigns have been proposed:1)Gateway Flowa. Gateway flow consists of two different entry points corresponding to MQ and Rest protocols.When a message is received in the gateway flow, a Unique-id is created using the integrationid and country code from the header. For this purpose, every message must contain theintegration-id and country code in the header.b. From the cache, the routing and transformation action details are retrieved using this UniqueId. These details consist of the integration-specific metadata for routing the message acrossthe framework. The Id is unique to each integration, not to a message. The routing details areappended to the header and the message is dynamically routed to the flow specified in therouting details.2)Processor Flowsa. This consists of a set of multiple process flows which are developed to fulfill differentintegration patterns. Depending on the complexity of the requirement, single or multipleprocess flows can be stacked together.b. Process flows fulfill the custom functionalities of each new requirement such as transformation,or any external REST calls. These flows can be run on separate containers to be deployableon-demand.c. Integration patterns like REST calls, transformation, record splitting, message routing acrossprotocols, etc. could be achieved and appended according to the process flows making thearchitecture more flexible and adaptable.d. Once the custom functionality of a requirement is executed, the message is routed to theflow as mentioned in the routing details.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 7
3)Dispatcher FlowDispatcher flow is the final exit point from the framework through which the output messageis transported to the target endpoint. The target endpoint is dynamically fetched from therouting details. All the integration-specific internal details and routing details are removedfrom the header before sending the message to the target endpoint.4)Cache Refresher Flowa. Cache refresher flow will run on specific time intervals and upload each scenario detail as keyvalue pairs onto the cache from time-to-time. Its main objective is to build a scenario configobject as a list of key value pairs and upload them to the cache.b. It fetches the existing list of scenario details and compares them with the updated detailsfrom DB and updates only the any new changes to the cache without disturbing unchangeddetails.c. A REST trigger is provided to refresh the cache instantly when changes are made to existingscenarios or new scenarios are configured.Here the key is the Unique-ID which will be used by the gateway flow to retrieve route detailsor metadata of the respective scenarios.Unique-ID Integration-ID Country-Code5)Audit and Exception Flowa. All the success messages that are obtained from the events will be captured, restructured,and inserted into the database by the Audit flow.b. These transactions shall be displayed by any custom analyzer.c. Exception flow will take care of error handling. It makes sure to capture every exception,restructure and categorize them accordingly. Any unhandled exception will be restructuredand logged into the unknown exception category.d. All the exceptions will be inserted into the database.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 8
6)Retry MechanismEach flow is carefully tailored to include a configurable retry mechanism to automaticallyreprocess the messages for failed transactions, which can be retried after a specific interval.7)Replaya. Exception-specific messages which can be stored and replayed by using the same transactionid previously stored in the database. This can only be triggered manually. Apart from the retrymechanism, replay is required in the case of necessary manual interventions to make thetransaction a success.b. Ex: Target endpoint is down for a while and all the messages that failed should be reprocessedonce the target is up and running.8)Persistent VolumeA persistent volume is mounted onto the required Pods to store the persistent data in theevent of a crash. When a new pod is started, the data is recovered from the persistent volumeand is processed without any data loss.9)CacheA Cache is used to store scenario-specific metadata or the routing details for runtimeidentification of incoming messages. This interaction happens once for every transaction inthe gateway flow. The cache used here represents a generic cache in place of which any typeof cache can be used.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 9
Benefits of using FrameworkMinimum number offlowsCollectiveData PersistenceDeploymentFulfills multipleintegrationrequirementsBenefits ofFrameworksReusableAvoids redeploymentfor each changeLoosely coupledCost-effectiveFigure 4 :Benefits of using FrameworkWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 10
High-Level ArchitectureThe following high-level architecture has an integration framework built on IBM ACE whichconnects the remaining integration applications like IBM MQ, IBM DP, and IBM APIC toexternal applications.Here all the remaining applications are used to showcase the connectivity of the frameworkwith the remaining applications. IBM DP and IBM API Connect are used as a gateway forthe framework. They are optional and can be replaced with any other on-cloud API gatewaypreferred by the client.Storage is also left to the discretion of the client. For more generic and cost-effective storageof the integration-specific configurations, a filesystem or on-cloud config-map can also beused.Audits can be logged by pushing the events/generated logs from pods to Splunk or anysimilar software. They can also be used to analyze each eSystemAPICMainframesFigure 5: HLAWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 11
Integration Patterns:Following are the integration patterns covered by this framework as of now.These patterns are categorized with the combinations of MQ and REST protocol-specifictransactions, which may or may not contain transformations or special functionalities.Each of the following pattern-specific descriptions explains how it is being handled withinthe current integration framework.Transactions from MQ to MQ:This category includes all the transactions that have the source and target endpoints asMessage Queues for the integration framework. Simple to complex requirements withthis nomenclature can be handled with similar configurationsThe flow diagram below shows the message transactions from MQ to MQ.MQGateway FlowCacheProcessor FlowDispatcher FlowRequest MessageIntegration LookupMessage, ConfigConfig Specific ProcessResultant Message, ConfigResultant Message, ConfigResponse MessageFigure 6: MQ to MQ TransactionsWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 12
Integration Patterns:Transactions from MQ to REST:These are all transactions in which the source is a Message Queue and the target is aREST endpoint for the integration framework. Simple to complex requirements withthis nomenclature can be handled with similar configuration as below.Following is the flow diagram for message transactions from MQ to REST.MQGateway FlowCacheProcessor FlowDispatcher FlowREST EndpointRequest MessageIntegrationLookupMessage, ConfigConfig SpecificProcessResultant Message,ConfigResultant Message,ConfigResponseMessageFigure 7: MQ to REST TransactionsWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 13
Integration Patterns:Transactions from REST to MQ endpoint:Transactions that have a REST endpoint as the source and a Message Queue asthe target are included in this category. Simple to Complex requirements with thisnomenclature can be handled with similar configurations as below.Following is the flow diagram for Message transactions from REST to MQ.RESTMQGateway FlowCacheProcessor FlowDispatcher FlowRequest MessageAcknowledgementIntegrationLookupMessage, ConfigConfig SpecificProcessResultant MessageResultant Message,ConfigResponse MessageFigure 8: REST to MQ TransactionsWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 14
Integration Patterns:Transactions from REST to REST endpoint:All transactions that have the source and the target as REST endpoints for theintegration framework, fall under this category. Simple to complex requirements withthis nomenclature can be handled with similar configurations. Here different REST toREST use-cases are explained.REST to REST use-cases:Framework as REST router to destinationAs soon as the message is received by the framework gateway flow, anacknowledgment is sent to the source, and the result of the message processing issent to the destination.RESTMQGateway FlowCacheProcessor FlowDispatcher FlowRequest MessageAcknowledgementIntegrationLookupMessage, ConfigConfig SpecificProcessResultant MessageResultant Message,ConfigResponse MessageFigure 9: REST to REST Transactions with AcknowledgmentWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 15
Integration Patterns:Framework as REST serviceThe resulting message after the transaction is completed is sent as a response tothe source.REST EndpointMQGateway FlowCacheProcessor FlowDispatcher FlowRequest MessageIntegrationLookupMessage, ConfigConfig SpecificProcessResultant MessageResultant Message,ConfigResponseMessageFigure 10: REST to REST TransactionsWhitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 16
IBM TechnologiesThe following technologies are used for representation purposes and to showcase theconnectivity with the framework built entirely on IBM App Connect.IBM APP ConnectIBM App Connect Enterprise combines the existing, industry-trustedtechnologies of IBM Integration Bus with IBM App Connect Professionaland with cloud native technologies. It delivers a platform that supportsthe full breadth of integration needs across a modern digital enterprise.IBM App Connect Enterprise software can be installed directly on aIBM ACEphysical machine running in your own Data Center, on a VMWare virtualmachine, in a Docker image, as part of an IBM Cloud Private installation,or installed by you into a public cloud such as IBM Cloud, AWS, or Microsoft Azure.IBM App Connect Enterprise enables information packaged as messages to flow betweendifferent business applications, ranging from large traditional systems to unmanned devicessuch as sensors on pipelines.In this architecture, IBM App Connect fills in the integration needs. In fact, the core of thisarchitecture is entirely built on IBM APP Connect. This architecture was designed to addressall possible combinations of requirements through one custom solution.A custom framework to run on clouds Using App Connect capabilities and a custom frameworkto run on clouds, the message is routed across servers using different protocols such asWeb Standards connectivity, REST connectivity, Cloud connectivity, ODBC connectivity, MQconnectivity, etc.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 17
IBM TechnologiesIBM API ConnectIBM API Connect is an end-to-end solution that allows users to create,secure, manage, socialize, monetize and analyze APIs. It provides apowerful set of capabilities from turning backend RESTFUL or SOAPservices into managed services. This is done by publishing APIs to APIGateways while enforcing lifecycle and governance controls on thoseAPIs. API Connect enables users to expose APIs, through a developerportal, targeting application developers both inside and outside theirorganization. Additionally, the solution’s analytics tooling helps APIproviders and API consumers better understand the health and consumption of deployedAPIs.IBM API Connect can be deployed in two patterns:1) Directly to a Kubernetes environment2) Appliance (OVA) install to an existing VMWare EstateKubernetes:API Connect can be installed to a Kubernetes environment that is running helm (includingtiller), ingress, and has a persistent storage solution.The APICUP utility enables the installation by creating a configuration file that is then usedwith the deployment.Appliance (OVA):The Appliances provide a base Kubernetes with helm, ingress, and local storage preconfigured.Each Appliance has the containers required for their component only i.e., the ManagementAppliance contains the Management Containers but not the Analytics containers.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 18
IBM TechnologiesIBM DB2The role of Database in this architecture is to: Contain the key-value pairs of all the integration-id and scenarioobject (consists of all the flows that will be executed for all interfaces)and populates them in the cache which will then feed them in theinternal header. Contain the audit, event, and error logs.The database picks up the message from the gateway flow and adds the key-value pair in theinternal header specific to the interface, thus routing it to the appropriate process flow for thebusiness integration to occur.The database also helps to store all the error, event and audit messages and hold them for anyreference later. These transaction details can be displayed by any custom analyzer.The maintenance process, however, should specify an expiration period after which the storedtransactional data will be deleted. The period the data will be alive can be decided by the clientsbased on their requirements and their criticality. Even the most critical data can be kept for along time and the less critical data can be deleted. We suggest having a maintenance flowwhich will delete the older messages, make way new messages, reduce storage, and increaseperformance.Here, database refers to IBM DB2 as our focus is on placing the IBM products on other clouds.This idea can also be extended to any other database like Oracle, SQL, etc. On-cloud databasescan also be used in case the client does not have DB2 and only needs one when moving otherIBM products to cloud.The choice of a traditional or cloud database is based on the specific business challenges andwhat’s most important to the client, along with performance, security, etc. If the need is tooffload many of the standard databases and infrastructure management tasks, then there area wide variety of options and cloud providers to choose from to get the advantages of clouddatabases. Managed storage for cloud databases is more likely your best choice if you require ahigher level of adaptability, scale, protection, and control over your database.As many cloud service providers have database migration services to help you move your datato the cloud, data migration should not be considered a problem if you decide to move to theon-cloud data services.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 19
IBM TechnologiesIBM MQIBM MQ is a family of message-oriented middleware products. As astateful application, it persists its state to storage for recovery in theevent of a restart.There are 3 choices available for HA:1. Native HA queue manager which has an active replica and twostandby replicas (available in MQ 9.2v)2. Multi-instance queue manager which is an active-standby pair, using a shared, networkedfile system.3. Single resilient queue manager running in a single Kubernetes Pod, where Kubernetesmonitors the queue manager and replaces the Pod as necessary.This offers a simple approach for HA using networked storage.Configuration changes on some volume fields can adjust persistence, including queues,messages, and recovery logs. The use of MQ in the framework allows us to work with messageswithout any data-loss and secure it at same time.Some default security features that MQ provides are:1. MQ channels provide the option to encrypt data in transit from applications and administratorsusing TLS connections. To enable TLS, the customer must make some configurations.2. A default TLS server certificate is configured for the queue manager signed by a publiccertificate authority such as Let’s Encrypt.3. Queue manager channels are configured by default with username/password authentication,which is backed by the IBM Cloud Identity and Access Management (IAM) for user andapplication management.4. All persisted queue manager data including messages, configuration, and logs, is encryptedat REST using full disk encryption on storage volumes.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 20
Continuous IntegrationContinuous integration is achieved through the usual process with the help of automatedjobs to build and deploy the virtual environment onto the cloud.The predefined jobs pull the updated image from the version control systems and build anddeploy them as containers on the cloud.IBM Stack On-cloudDeploymentAutomationVirtual EnvironmentSetupVersion Control SystemOn-PremisesFigure 11: Continuous Integration RepresentationConclusionNecessity is the mother of invention. Although it is necessary to bring the integration segmentonto the cloud platform in Digital Transformation era, it is important that it is being donewithout compromising the quality, response time, and performance. The proposed businesssolution has been designed keeping in mind all the aspects of the integration and ensuresthat the reliability, efficacy, efficiency, and security are maintained. It is high time that theorganizations pick up the integration solutions that are adaptable to all clouds thus making itflexible and cost-effective along with its other benefits.Whitepaper Container-based Framework to Migrate Disparate IBM Integration Patterns on Any Cloud 21
References :IBM APP .0.0?topic //www.ibm.com/docs/en/app-connect/11.0.0?topic overview-app-connect-enterprise-technicalIBM APIC:https://www.ibm.com/docs/en/acfc?topic api-connect-cloud-overviewIBM s/content/SSEPGG 11.5.0/com.ibm.db2.luw.wn.doc/doc/c0060311.htmlIBM MQ:https://www.ibm.com/docs/en/ibm-mq/9.2?topic d.ibm.com/docs/mqcloud?topic mqcloud-mqoc www.ibm.com/docs/en/ibm-mq/9.2?topic containers-high-availability-mq-inOther-References ntainersLTI (NSE: LTI) is a global technology consulting and digital solutions Company helping more than 475 clients succeedin a converging world. With operations in 33 countries, we go the extra mile for our clients and accelerate their digitaltransformation journeys. Founded in 1997 as a subsidiary of Larsen & Toubro Limited, our unique heritage gives us unrivalledreal-world expertise to solve the most complex challenges of enterprises across all industries. Each day, our team ofmore than 40,000 LTItes enable our clients to improve the effectiveness of their business and technology operations anddeliver value to their customers, employees and shareholders. Find more at http://www.Lntinfotech.com or follow us at@LTI Global.info@lntinfotech.com
like IBM MQ and IBM Integration Bus. It has provided IBM Cloud Pak for integration on IBM-Cloud and with RedHat OpenShift to run this Cloud Pak on other third-party clouds without compromising any of its features. As everything is now available for integration technologies to run on any cloud, a digital integration transformation framework is .
container container container container container networking storage registry security logs & metrics container orchestration & cluster management (kubernetes) fedora / centos / red hat enterprise linux container runtime & packaging (docker) atomic host infrastructure automation & cockpit
container container container container container networking storage registry security logs & metrics container orchestration & cluster management (kubernetes) fedora / centos / red hat enterprise linux container runtime & packaging (docker) atomic host infrastructure automation & cockpit
Migration support Can migrate from Cisco Unity Connection 1.x Can migrate from Cisco Unity 4.0(5) or later Can migrate from VPIM compatible VM systems Standalone or Digitally Networked Linux-based appliance Optional integration with Exchange and/or Active Directory No IBM Lotus Domino or MS Exchange experience required for VM admin and maintenance
People migrate when they move from their home to another home in a different place. Plants migrate over time as the climate changes. They move to an area that better fits their needs. Animals migrate with the seasons from one region to another. ANIMAL MIGRATION . Animals that migrate are called . migratory animals. or migrants.
2008 komatsu wa430-6 wheel loader 2013 envirotank 73000 litre skid mounted steel fuel tank 2001 jindo 48 ft high cube container 2003 cimc 40 ft container 2005 quingdao 40 ft container 1996 changzou 20 ft container 2002 evergreen 20 ft container jindo 20 ft storage container 2001 alta-fab wellsite 2010sentag 12 ft x 60 ft 3 unit skid mounted .
Oracle Container Runtime for Docker 19.03 1-2 Oracle Container Runtime for Docker 18.09 1-3 Oracle Container Runtime for Docker 18.03 1-3 Oracle Container Runtime for Docker 17.06 1-4 Docker 17.03 1-5 Docker 1.12 1-6 2 Installing Oracle Container Runtime for Docker Setting Up the Unbreakable Enterprise Kernel 2-1
B.A.G. CORP. SUPER SACK CONTAINER CATALOG S U P E R S A C K C O N T A I N E R D E S I G N S S U P E R S A C K C O N T A I N E R D E S I G N S 7 Spread Strap container Tubular Super Sack container Four-Panel Super Sack container Hardwall container in open position. Barrel Bag container LINER OPTIONS ARE AVAILABLE FOR ALL OF OUR FIBCS .
THE CONTAINER SHIP Task 1: Word spider - write down as many terms you know about the design and construction of the container ship TEU cell guide The main characteristic of a container ship is that it depends on shore-based lift-on/lift-off equipment, mainly container gantry cranes (also called portainers), to handle its cargo. The earliest .
