Enterprise Integration Patterns:Enterprise Integration Patterns: Past .

Enterprise Integration Patterns: Past, Present and Future Translator Test Message Splitter Aggregator Enricher Gregor Hohpe www.eaipatterns.com

2011 Gregor Hohpe

Isolated Systems Unified Access

70s: Batch Data Exchange Export information into a common file format, read into the target system Example: COBOL Flat files System A E x p o r t Customer C t Data Pros: Good physical decoupling Language and system independent I m p o r t System B Database Cons: Data transfer not immediate Systems may be out of sync Large amounts of data

80s: Central Database All applications access a common database System A System B System C Customer Data Pros: Consistent Data Reporting Transactional guarantees Cons: Integration of data, not business functions Difficult to find common representation

90s: Remote Procedure Calls One application calls another directly to perform a function. Data necessary for the call is passed along. Results are returned to calling application. System A Pros: Data exchanged only as needed Integration of business function, not just data Get Credit Score 740 System B Cons: Works well only with small number of systems Fragile (tight coupling) Performance

Asynchronous Messaging Style Sender Receiver Message Channel (Q (Queue) ) Systems send messages across Channels Channels have logical (location-indep.) (location indep ) addresses g a message g into the Channel is Placing quick (“fire-and-forget”) The Channel queues messages until the receiving application is ready Simplified Interaction Location Decoupling Temporal Decoupling An "honest" architectural style that does not try to deny the limitations of the underlying medium.

Why Asynchronous Messaging? Asynchrony Sender does not have to wait for receiver to process message Temporal decoupling Throttling Receiver can consume messages at its own pace Processing units can be tuned independently Can be Reliable Over Unreliable Networks Messages can transparently be re-sent re sent until delivered Think cell phones – intermittent and unreliable Insertion of intermediaries (Pipes-and-Filters) Composability Transformation, routing etc. Throughput over latency “Wider bridges not faster cars”

ab bility to e execute A New “Tower of Babel” completeness of vision Gartner “Magic Quadrant” for Integration and Middleware 2001

Messaging Pattern Language 1. Transport messages Application Application Application Application 2. Design messages 3. Route the message to the proper destination 4. Transform the message to the required format 5. Produce and consume messages 6. Manage and Test the S t System Application Application

Messaging Pattern Language 1. Transport messages Channel Patterns 2. Design messages Message Patterns 3. Route the message to the proper destination Routing Patterns 4. Transform the message to the required format Transformation Patterns 5. Produce and consume messages Endpoint Patterns 6. Manage and Test the S t System Management Patterns Application

http://hillside.net/ plop/plop2002/ proceedings.html “Enterprise Integration Patterns” G. Hohpe “Patterns of System Integration with Enterprise Messaging” B. Woolf, K. Brown 2006 Gregor Hohpe

OOPSLA 2003 185,000 Words 700 pages 50,000 copies Translations English Russian Chinese Traditional Korean www.eaipatterns.com Sketches, summaries under Creative Commons Visio, Omnigraffle stencils

Message R o u tin g M es s a g e C o n s tru c tio n M es s age Router Content-B as ed Router M es s age Filt Filter Dy nam ic Filter Rec ipient Lis t S plitter A ggregator Res equenc er Com pos ed M s gg. P roc ess or or. S c atter-Gather Routing S lip P roc es s M anager M es s age B rok er M es s age Com m and M es s age Doc um ent M es s age E vent M es s age Reques t-Reply Return A ddres s Correlation Identifier M es s age S equenc e M es s age E x piration Form at Indic ator M e ss a g e T ra n s fo rm a tio n M es s age Trans lator E nv elope W rapper Content E nricher Content Filter Claim Chec k Norm aliz er Canonic al Data M odel Endpoint p Endpoint p M essage Channel Router Translator A pplication A M e s s a g in g E n d p o in ts M es s age E ndpoint M es s aging Gateway M es s aging M apper Trans ac tional Client P olling Cons um er E v ent-Driv en Cons um er Com peting Cons um ers M es s age Dis patc her S elec tiv e Cons um er Durable S ubs c riber M es s aging A dapter Idem potent Rec eiv er S erv ice A c tiv ator A pplication B M e s sa g in g C h a n n e ls M es s age Channel P oint-to-P oint Channel P ublis h-S ubc r. Channel Dataty pe Channel Invalid M es s age Channel Dead Letter Channel Guaranteed M es s aging p Channel A dapter M es s aging B ridge M es s age B us M onitoring g S y s te m s M a n a g em en t Control C t l B us Detour W ire T ap M es s age His tory M es s age S tore S m art P rox y Tes t M es s age Channel P urger

Visual Language Content-Based Router Message Filter Recipient List Splitter A Aggregator t Resequencer Routing Slip (Itinerary) Process Manager

Composing Patterns Receive an order Get best offer for each item from vendors Combine C bi into i t validated lid t d order. d Scatter-Gather Quote Pub-Sub Channel Vendor A Vendor B New Order Validated Order Splitter Aggregator Quote Request for each item “Best” Best Quote for each item Vendor C A Aggregator t

Someone better build a tool that supports these! Home-made wisdom elixir 2011 Gregor Hohpe

Tossers. II’llll just do it myself! Cheers! 2006 Gregor Hohpe 20

2006 Gregor Hohpe

2006 Gregor Hohpe

Patterns Origin – Christopher Alexander BED ALCOVE Design problem Bedrooms make no sense. Forces Fi First, the h bed b d in i a bedroom b d creates awkward k d spaces around it: dressing, working, watching television, sitting, are all rather foreign to the side spaces left over around a bed bed. ((.)) Second, the bed itself seems more comfortable in a space that is adjusted to it. Solution Don't put single beds in empty rooms called bedrooms, but instead p put individual bed alcoves off rooms with other nonsleeping functions, so the bed itself becomes a tiny private haven.

Patterns – 15 Years After GoF “Mind sized” chunks of information (Ward Cunningham) Human-to-human interaction Expresses intent (“why” vs. “how”) Observed from actual experience New programming models bring new patterns NOT: A firm rule –always a time when not to use Copy-paste code Isolated. Part of a Pattern Language

Patterns as Executable Domain Language? CUSTOMER ENRICHER SPLITTER ROUTER LOGGER LOGGER orderChannel orderChannel orderEnrichedChannel orderEnrichedChannel itemChannel "/Order/Item“ / / itemChannel coldBevChannel "Item 'FRAPPUCINO'“ hotBevChannel coldBevChannel hotBevChannel Logger Customer Enricher Splitter Router coldBevChannel Logger order Ch Channel l orderEnriched Ch Channel l item Ch Channel l hotBevChannel

Patterns Components

Patterns Components Human communication Fuzzyy Design tool p Platform independent System Communication Precise Executable Platform dependent p Pipes and Filters style: simple composability Easily formalizable: input ports, ports output ports Input Port Output Port Leverage other domain languages, such as XSLT

Enterprise Integration or Messaging? Enterprise Integration Messaging

Enterprise Integration or Messaging? Enterprise Integration Messaging Conversations Processes Events

Messaging Flow of messages through processing nodes Translator T tM Test Message S litt Splitter A Aggregator t Enricher Stateless - scaleable, decoupled Error handling? Complex interactions (no guarantees)

Conversations Conversation C ti State Internal State: P Processing i Payment Order Invoice Payment Drinks Internall State: Waiting for Payment Internal State: Making Drinks Each conversation corresponds to one process instance Each participant has a (potentially different) process definition

Simple Example: Request Response Client Server Conversation Series of Related Messages

Response Message Lost Client Server

Response Message Delayed Client Server

Conversation State Client Server Waiting for Response

Challenges: Describing Conversations Sequence Diagrams (UML 1.x) only show one instance, not rules of interaction Sequence Diagrams (UML 2.0) more powerful, but non-intuitive notation WS-CDL describes conversation state, but very little adoption p WS-BPEL a little verbose, looking from participant perspective Temporal Logic expressive, but not good for sketch

Request-Reply Conversation State Chart Request Channel Requestor Reply Channel Awaiting Answer Replier Simplest conversation Single Conversation state: waiting for reply, complete Gets G more complicated li d once error conditions di i considered id d 37

Request-Reply with Retry Conversation C ti State t Req est Request Awaiting Answer Request (Resend) [timeout] Response Consumer max retry? Provider [[response] p ] [yes] Sender can repeat request n times Success Failed Provider has to be idempotent Receiver also has to be idempotent Example: RosettaNet Implementation Framework (RNIF)

How can a service find a conversation partner in when it has no knowledge about the network and its services? Point-to-point p communication requires q knowledge g of the conversation partner (or channel). The late binding between a service consumer and the service endpoint p lowers the location coupling p g between them. Discovery may be on the critical path to establishing a conversation. Even in the presence of a central lookup service, a new participant has to first establish a connection to the lookup service.

Dynamic Discovery Request 1 1. 1 2. 3. 4 4. 5. Pub-Sub 4 Choose 5 Interact Consider 2 Provider 1 3 Provider 2 Respond Provider 3 Broadcast request Provider(s) consider whether to respond (load, suitability) Interested providers send responses R Requestor t chooses h “best” “b t” provider id from f responses Requestor initiates interaction with chosen provider Examples: DHCP, TIBCO Repository discovery

How can a service find a conversation partner across a large p g network without flooding the network with requests? q The late binding g between a service consumer and the service endpoint lowers the location coupling between them. Discovery may be on the critical path to establishing a conversation. Many networks do not route broadcast packets beyond the local network. Often centralized administration is involved in setting up a new service.

Consult Directory 2 Consult Directory Service Registry 1 Consumer Conversation 3 Publish Service Contract Provider Directory may store additional metadata about the service "Match making based on" Unique Identifiers Interface Definition / Type Attributes Keyword y match

Subscribe-Notify (Multi-responses) Express Interest N tif Notify Subscriber Provider Subscriber expresses interest in receiving notifications Subscriber receives messages until a stop condition is reached: Subscriber sends a stop request A deadline is reached without the subscriber renewing interest Subscriber does not respond to requests from provider Provider notifies subscriber of end of transmission

Renewing Interest Automatic Expiration t Register Lease (Renew Interval) Renew Interest Subscriber Provider “Lease” model Heartbeat / keep keep-alive alive Subscriber has to renew activelyy Example: Jini Renewal Request Register Renewal Request Renewal Confirm Subscriber t Provider “Magazine Model” Subscriber can be simple Provider P id has h to t manage state for each subscriber

Conversation Pattern Language Messages Initiating Message Follow-on Message Complete Message Side Conversation (Sub-Conversation) (Sub Conversation) Acknowledgment Message Simple Conversations Reliable Delivery Sync Request-reply Async. Async Request Request-Reply Reply messages Async. Request Request-Poll Poll for result Subscribe Subscribe-Notify Notify Tacit Agreement Reaching agreement