A Reactive Performance Monitoring Framework
A Reactive PerformanceMonitoring FrameworkKatherine ChengLiDirected By:Prof. Liam PeytonThesisSubmitted to theFaculty of Graduate and Postdoctoral Studiesin partial fulfillment of the requirements for the degree ofMaster of Computer ScienceUniversity of OttawaOttawa, Ontario, CanadaJune 2016 Katherine ChengLi, Ottawa, Canada, 2016
AbstractWith the ascendency of data and the rise of interest in analytics, organizations arebecoming more interested in the use of data to make their business processes moreintelligent and reactive. BI applications are a common way that organizations integrateanalytics in their processes. However, it can be days, weeks or even months before amanual response is undertaken based on a human interpreting a report. Even wheninformation technology supports automatic responses within an organization, it is oftenimplemented in an ad hoc manner without following a systematic framework.In this thesis, we present a reactive performance monitoring (RPM) frameworkwhich aims at automating the link from the analytical (how well is the operationalachieving the strategic) to the operational (the particular process steps implementedwithin an organization that determine its behavior) aspects of businesses to bypass thestrategic (the high level and long term goals an organization is trying to achieve) asneeded and reduce the latency between knowledge and action. Our RPM framework iscomposed of an architecture, a methodology, and a rule environment which permits theredaction of rules possessing relevant conditions and actions. In addition, we present anOLAP rule engine which is demonstrated to be effective in our framework where eventsare streamed in, reacted upon in real-time, and stored in an OLAP database.To develop and evaluate our framework, two case studies were undertaken. Thefirst was done using IBM technologies implementing an application made to identifypatients at high risk of cancer recurrence. The second was done using open sourcetechnologies. With this second implementation, we created an application that has theii
goal of informing women from at risk populations of the different stages of pregnancy ona weekly basis.iii
AcknowledgementsAlthough the accomplishment of a thesis is a personal achievement, I wouldcertainly not have been able to finish it without the help, support, and patience of mysupervisor Prof. Liam Peyton. Thank you for your continued help and encouragement.Thank you to my family for encouraging me throughout this process and forbelieving that I would be able to successfully accomplish this. A special thank you toMathieu Jobin for his never ending support.In addition, I would like to thank the organizations that helped fund my master’sdegree. This includes MITACS, NSERC, and OGS.Finally, thank you to IBM and CENGN for providing me the opportunity to internwith them. In particular, thank you to Randy Giffen from IBM and my Extreme Blueteam. It has been a great source of knowledge and experience.iv
Table of ContentsAbstract . iiAcknowledgements . ivTable of Contents . vList of Figures . viiiList of Tables . ixList of Acronyms . x12Introduction .11.1Problem Statement .11.2Thesis Motivation and Contributions .31.3Thesis Methodology and Organization.5Background .92.1Performance Monitoring .92.1.1Performance Management .92.1.2Real-time Analytics . 102.1.3Predictive Analytics . 112.1.4Event . 112.2Business Process Management . 112.2.1Business Process . 112.2.2Web Service . 122.2.3Business Process Management (BPM) . 122.3Event Processing Technologies. 142.3.1Message Broker . 152.3.2Event-Driven Architecture . 152.3.3Complex Event Processing (CEP) . 152.3.4Notification . 162.4On-Line Analytical Processing (OLAP). 162.4.1Data Access Object . 162.4.2Business Intelligence (BI) . 172.4.3Data Warehouse . 17v
2.4.4Star Schema . 182.4.5Online Analytical Processing (OLAP). 182.5Rule Engine . 192.5.1State Monitoring Engine (SME) . 192.5.2IBM SPSS Analytical Decision Management (IBM SPSS ADM) and IBM SPSSModeler 202.5.32.63Related Works . 212.6.1Application Framework for Monitoring Care Processes (AFMCP) . 212.6.2Big data Analysis Infrastructure Testbed (BAIT) . 222.6.3IBM Predictive Maintenance and Quality (IBM PMQ). 25A Reactive Performance Monitoring Framework . 263.1Problem Description . 263.2Gap Analysis. 293.3Evaluation Criteria . 313.3.1Framework. 313.3.2Rule Environment . 333.4Overview of the Reactive Performance Monitoring Framework . 353.4.13.4.1.14Drools and IBM Operational Decision Manager . 20Architectural Pattern . 36Critical Interfaces . 413.4.2Methodology . 413.4.3Rule Environment . 493.4.4OLAP Rule Engine . 51Case Studies . 554.1Overview. 554.2IBM PMQ . 574.2.1Architecture . 574.2.2Results . 614.3IBM RPM . 624.3.1Architecture . 624.3.2Methodology . 644.3.3Rule Environment . 66vi
4.3.44.456Results . 67QuickForms RPM . 674.4.1Architecture . 694.4.2Methodology . 714.4.3Rule-Based Environment . 724.4.4Results . 744.5Rule Example 1 . 744.6Rule Example 2 . 76Evaluation . 795.1Comparison of Frameworks . 795.2Comparison of Rule Engines . 845.3Limitations and Assumptions . 87Conclusions and Future work . 896.1Conclusions . 896.2Future Work. 90References . 91vii
List of FiguresFigure 1: DSRM (Design Science Research Methodology) Process Model (Peffers et al., 2007) .5Figure 2: The basic elements of performance management (Pollitt, 2013) . 10Figure 3: BPM Life Cycle as described by (Pourshahid et al., 2009) . 13Figure 4: The BPM life cycle consisting of three phases: (1) (re)design, (2) implement/configure,and (3) run and adjust. (van der Aalst, 2013) . 14Figure 5: Data Access Object (Sun Microsystems Inc., 2002) . 17Figure 6: Example usage of AFMCP (Aladdin Baarah, 2013) . 22Figure 7: BAIT Architecture Layers (Kelly et al., 2015) . 23Figure 8: Response Gap . 27Figure 9: RPM Framework Architecture . 37Figure 10: RPM Framework Methodology . 43Figure 11: Rule Environment . 50Figure 12: OLAP Rule Engine Object Model . 51Figure 13: Simplified architecture view of IBM PMQ . 58Figure 14: IBM PMQ Master Data Model (IBM Corporation, 2016a) . 59Figure 15: Example of a simple Orchestration file . 61Figure 16: IBM RPM Architecture . 63Figure 17: QuickForms Application Framework (QuickForms (uOttawa), 2015a) . 68Figure 18: QuickForms RPM Architecture . 70viii
List of TablesTable 1: WeeklyDueDateReport example . 76Table 2: FemaleAtRiskReport example. 78Table 3: Comparison of Frameworks. 80Table 4: Comparison of Rule Engines . 85ix
List of AcronymsxAcronymDefinitionAFMCPAJAXApplication Framework for Monitoring CareProcessesAsynchronous JavaScript and XMLAPIApplication Programming InterfaceASDIAircraft Situation Display to IndustryBAITBig data Analytics Infrastructure TestbedBIBusiness IntelligenceBPELBusiness Process Execution LanguageBPMBusiness Process ManagementBPMNBusiness Process Model and NotationCEPComplex Event ProcessingCIWSCorridor Integrated Weather SystemCoSPAConsolidated Storm Prediction for AviationCPMACare Process Monitoring ApplicationCPMECare Process Monitoring EngineDAOData Access ObjectDSRMDesign Science Research MethodologyETEcho TopETLExtract Transform LoadGUIGraphical User InterfaceHTMLHypertext Markup Language
xiHRRRHigh-Resolution Rapid RefreshIBM ODMIBM Operational Decision ManagerIBM PMQIBM Predictive Maintenance and QualityIBM SPSS ADMIBM SPSS Analytical Decision ManagementIBM SPSS C&DJSONIBM SPSS Collaboration and DeploymentServicesJavaScript Object NotationJVMJava Virtual MachineKPIKey Performance IndicatorLDMLocal Data ManagerOLAPOnline Analytical ProcessingRESTRepresentational State TransferRPMReactive Performance MonitoringRTLSReal-Time Location SystemSMEState Monitoring EngineSOAPSimple Object Access ProtocolVILVertically Integrated LiquidXMLExtensible Markup Language
1 Introduction1.1 Problem StatementWith the ascendency of data and the rise of interest in analytics (Brown, 2015),organizations are becoming more interested in the use of data to make their businessprocesses more intelligent and reactive. Traditionally, organizations define operationalbusiness processes to achieve their strategic business goals (improve revenues, reducecosts, ensure customer satisfaction, improve quality of results), and then they collect datafrom those operational business processes into data warehouses where reports are createdand the results are analyzed using Business Intelligence (BI) applications to see if theoperational business processes are effective in achieving the strategic business goals.Unfortunately, the creation of such BI applications is complex.Traditionally, BI applications follow Kimball’s (Kimball, Ross, Thornthwaite,Mundy, & Becker, 2008) methodology: the requirements are initially collected; followedby the technical architecture and its plan; then design of the dimensional model; thenphysical and performance database planning; then design and development of the ETLsystem; finally the design and the development of the BI applications. However, there islittle direct linkage between the analysis done in these applications, and appropriatereactions to adjust business processes in response. It can be days, weeks or even monthsbefore a manual response is undertaken based on a human interpreting a report. Evenwhen information technology supports automatic responses within an organization, they1
are often implemented in an ad hoc customized manner without following a systematicframework.As Chamney described in his thesis (Chamney, 2014), the creation of BIapplications and the accompanying backend is non-trivial. If we also add complex eventprocessing (CEP), or business process management (BPM) to achieve some degree ofautomation, it can only be imagined that the creation of a seemingly simple applicationcan quickly become complex without careful and controlled planning and execution.The literature does offer examples of use cases where some sort of analyticsframework has been implemented to achieve specific goals. For example, (Kelly, Craig,& Matthews, 2015) created an architecture to provide “analytics-as-a-service foralgorithm developers to use when creating and testing new capabilities for air trafficmanagement decision support.” Although these different architectures provide aframework for building BI applications they often do not possess a rule environment todirectly connect reports and decisions to actions based on incoming events and existingknown variables.This illustrates a lack of bridging between the operational (the detailed steps wefollow as a business), and the analytical (how well is the business fulfilling the strategic)aspects of businesses. Typically, feedback is unidirectional with insufficient context.Moreover, feedback is given by people after the fact. This creates latency between havingthe data that informs us of what needs to be done and actually doing what needs to bedone. The use of data for analytics is very disconnected from operational use so that2
determining reports from operational data is done in an ad hoc manner, and determiningoperational actions from reports is also done in an ad hoc manner.1.2 Thesis Motivation and ContributionsThe main goal of this thesis is to provide a framework which will enableorganizations to systematically design the relationships and reactions between theanalytical and operational parts of business so that businesses are not simply limited toperformance management of their operations to determine if strategic goals are achieved,but rather can define reactive performance monitoring so that automatic actions can betriggered operationally in real-time when needed as indicated by analytics.Technology in the form of business process management exists to automatebusiness processes (van der Aalst, 2013); and technology exists in the form of predictivemodelling (zur Mühlen & Shapiro, 2010) to analyze data and predict when actions areneeded; and technology exists for complex event processing that can stream data forprocessing and define rules to trigger actions when determined by a model (Luckham,2012).It was our initial motivation to combine these technologies in a systematicframework.Most rule engines used in complex event processing are based on simple objectmodels that are independent of the rich multidimensional OLAP databases that are usedin Business Intelligence applications (Abdelfattah, 2013). Another motivation for ourwork is that more appropriate rule engines and an architecture optimized for reactiveperformance monitoring could be created if OLAP Databases were more often integratedwith complex event processing (Costa, Cecílio, Martins, & Furtado, 2011).3
This thesis proposes a Reactive Performance Monitoring (RPM) framework thatprocesses and analyzes events to determine appropriate re-actions across systems that areneeded in a scenario where data is gathered, reported on, and acted upon or directed asneeded. In particular, RPM is based on a rule environment which can trigger meaningfulactions methods with condition functions based on analytics. RPM rests on a datawarehousing infrastructure centered around OLAP databases. Ultimately, RPM attemptsto close the gap between the strategic, analytical, and operational parts of a business byproviding relevant interfaces and feedback mechanisms.The main contributions of this thesis are the following:1. A reactive performance monitoring framework (RPM) including:a. An architecture pattern which integrates various technologies andstructures the event flowb. A methodology to build applications using said frameworkc. A rule environment using appropriate condition functions andaction methods to respond to operational events based on data froman analytical data warehouse2. An OLAP Rule EngineWhile working on this thesis we were fortunate to be able to collaborate with twodifferent organizations on two different internships. The first one is IBM (IBMCorporation, 2016c), the multinational technology company, where we were able to learnmore about their technologies used in our case studies. The second internship was atCENGN (CENGN, 2016). CENGN is a network technologies accelerator and consortium4
where we were able to participate in both their projects and underlying infrastructure. Wealso have one publication so far from this thesis:ChengLi, Katherine. (2014, November) Smarter Process Triggers: Predictive Analyticsfor Smarter Business Operations. Poster session presented at CASCON 2014 24th AnnualInternational Conference on Computer Science and Software Engineering “SmarterSystems of Interactions”, Toronto, Canada. (ChengLi, 2014)1.3 Thesis Methodology and OrganizationWe followed the Design Science Research Methodology (DSRM). As describedby (Hevner, March, Park, & Pam, 2004), DSRM is adapted to situations where thecreation of an artifact is “intended to solve identified organizational problems”. Figure 1describes the entire process model of the DSRM proposed by (Peffers, Tuunanen,Rothenberger, & Chatterjee, 2007). This is the process we followed in the seconditeration. For our first iteration, although still following DSRM, the process followed wasa bit less structured.Figure 1: DSRM (Design Science Research Methodology) Process Model (Peffers et al., 2007)5
1. Identifying the problem: A problem was identified with the integration ofpredictive analytics in business processes to bridge the gap between eventbased analytics triggers and business processes. The problem wasidentified through the implementation of a proof of concept with IBMtechnologies at IBM and the comparison of their solution with currentresearch group activities.2. Defining the objectives of the solution: The goal of was to reduce the gapbetween business processes and real-time events.3. Designing and developing the artifact: To reduce this gap, a frameworkintegrating real-time Fact Events to trigger business processes wasdesigned.4. Demonstrating the artifact: In this first iteration, the initial problem wascontext initiated. Our demonstration consisted of a proof of concept madewith IBM technologies.5. Evaluating the artifact: We evaluated the framework comparing it withcurrent practices and with the use of predictive analytics only.6. Communication: A poster presentation was made called “Smarter ProcessTriggers: Predictive Analytics for Smarter Business Operations.”For our second iteration, these steps were followed:1. Identifying the problem: DSRM as defined by (Peffers et al., 2007) startswith the identification of the problem and its definition through which theimportance of the problem is shown. In our case, we re-identified the6
problem. Businesses try to integrate analytics or big data in their day today operations but with much difficulty. There is a considerable gapbetween the strategic and the operational when trying to integrate theresults of analytics in real-time. We adapted our initial problem related toFact Event based triggers for business processes to a broader analyticaloperational bridging problem.2. Defining the objectives of the solution: The second step of DSRM consistsof defining the objectives of the solution and defining how the resultingartifact would yield better results. We want to bridge the gap between thestrategic, the analytical, and the operational.3. Designing and developing the artifact: We propose a framework whichintegrates several technologies in a structured way with defined interfacesand a methodology guiding the construction of the architecture and theintegration of each part.4. Demonstrating the artifact: To demonstrate our framework, we have twoexample uses of the framework. One uses IBM technologies, and the otheruses open source technologies, namely QuickForms. These two exampleshelped refine the architecture and identify the existing gaps moreprecisely.5. Evaluating the artifact: The framework was evaluated, validated, andreviewed by comparing it with literature examples as well as currentpractices, and the use of a data warehouse only. Our criteria are based on7
the evaluation of the gaps between strategic, analytical, and operationalparts of a business.6. Communication: This thesis presents our final results.As it is usually done in DSRM, our framework was actually re-evaluated multipletimes in micro-iterations to be redefined and redesigned.The thesis is organized as follows. Chapter 1 introduces the thesis. In Chapter 2,we present the underlying concepts needed to understand this thesis as well as the relatedworks. In Chapter 3, we present the problem, our evaluation criteria, and our framework.In Chapter 4, two case studies and some rule examples are presented. In Chapter 5, ourevaluation of the framework is given. Chapter 6 concludes our thesis.8
2 BackgroundFor a better understanding of this thesis, there are several concepts that should beknown beforehand. This chapter is dedicated to those concepts as well as related worksthat have tried to solve or partially solve the problem stated in Chapter 1 Introduction.We have separated this chapter in 5 sections: Performance Monitoring, Business ProcessManagement, Data Integration Technologies, On-Line Analytical Processing, and RelatedWorks.2.1 Performance MonitoringTo understand RPM, our reactive performance monitoring framework, it isnecessary to understand performance monitoring. This section is dedicated to relatedconcepts. We outline performance management, real-time analytics, predictive analytics,and the concept of an event.2.1.1 Performance ManagementAs described by (Pollitt, 2013), performance management is composed of 6 maincyclical elements:9 activities (in business processes) measurements of particular aspects of activities (metrics) data criteria against which to compare the data, relevant information obtained from the data,
and decision making pertaining to the activities (which leads to newactivities)Figure 2: The basic elements of performance management (Pollitt, 2013)In short, it is used to measure and communicate the success (or lack thereof) ofbusiness processes activities using metrics (Middleton, Peyton, Kuziemsky, & Eze,2009).2.1.2 Real-time AnalyticsReal-time analytics consists of events, or simply data, being stored in real-timeand displayed in real-time via a medium such as a dashboard, metrics, or alerts. Thedisplay then consists of current and historical data (Chieu & Zeng, 2008). Organizationalperformance can be measured using real-time analytics comparing fine-grained real-timeperformance metrics to actual objectives (Mouttham, Peyton, & Kuziemsky, 2011).10
2.1.3 Predictive Analytics“Predictive analytics uses statistical techniques, machine learning, and datamining to discover facts in order to make predictions about unknown future events”(Lechevalier, Narayanan, & Rachuri, 2014). As one of the advantages, this can allow thebusiness to act ahead of time and diminish costs because actions have been undertakenbefore the system actually fails (Dash, Giffen, Ramakrishnarao, & Sreenivasan, 2013).Predictive analytics lets you predict and act ah
IBM ODM IBM Operational Decision Manager IBM PMQ IBM Predictive Maintenance and Quality IBM SPSS ADM IBM SPSS Analytical Decision Management IBM SPSS C&D IBM SPSS Collaboration and Deployment Services JSON JavaScript Object Notation JVM Java Virtual Mac
The number of animals with serum antibodies that were reactive, moderately reactive, or non-reactive against each of the pathogens investigated. Item EHV-1 EIV WNV Lyme Reactive 3 0 0 6 Moderately Reactive 29 9 0 1 Non-Reactive 60 61 92 85 A total of 92 serum samples were tested in the EHV-1 risk evaluation assay. In total, 3% (3/92)
A permeable reactive barrier can be designed as (1) a continuous wall containing reactive material or (2) a funnel-and-gate system, where the funnel consists of two impermeable walls that direct groundwater flow to the central reactive-material gate. Permeable reactive barriers have been used in a number of
Keywords Functional Reactive Programming, Graphical User In-terfaces 1. Introduction Elm is a functional reactive programming language that aims to simplify the creation of responsive graphical user interfaces (GUIs), and specifically targets GUIs for web applications. Func-tional reactive programming
Nov 19, 2015 · Alkali Metals 1 s1 ending Very reactive Alkaline Earth Metals 2 s2 ending Reactive Transition Metals 3-12 (d block) ns2, (n-1)d ending Somewhat reactive, typical metals Inner Transition Metals f block ns2, (n-2)f ending Somewhat reactive, radioactive Halogens 17 s2p5 endi
The Reactive Sample Project A RESTful master-detail web application that communicates to a local REST API using json-server A reactive master-detail web application that uses @ngrx/store We will be making the widgets feature reactive Feel free to use the existing code as a reference point
telemetry 1.24 Service P threshold_migrator 2.11 Monitoring P tomcat 1.30 Monitoring P trellis 20.30 Service P udm_manager 20.30 Service P url_response 4.52 Monitoring P usage_metering 9.28 Monitoring vCloud 2.04 Monitoring P vmax 1.44 Monitoring P vmware 7.15 Monitoring P vnxe_monitor 1.03 Monitoring vplex 1.01 Monitoring P wasp 20.30 UMP P .
What is Media Monitoring and How Do You Use it Monitoring: a history of tracking media What is monitoring? Getting started with monitoring The Benefits and Uses of Monitoring Using media monitoring to combat information overload Tools to maximize monitoring and measurement efforts Using media monitoring to develop media lists
System Monitor - Performance Monitoring Services 5.5 (hereafter referred to as "System Monitor - Performance Monitoring Services") is a software program that provides a simple means of monitoring the performance of multiple machines. This manual explains how to use System Monitor - Performance Monitoring Services. This manual is organized as .
