Cask Data Application Platform (CDAP)

Deployment ArchitectureThis section describes different components of the CDAP runtime system and how they are deployed on aHadoop cluster. For more information please see hereThe CDAP runtime system is made up of two major components:CDAP ServerThe CDAP Server is a collection of services (Figure 2) essential for successfully running CDAP. It can beinstalled on one or more edge nodes of a cluster and is responsible for managing only the cluster to whichit’s configured. The services are installed for a few reasons:Fixed IP/Hostname for accessing REST APIsImpersonating a user in secure mode to run in a clusterManage (Start/Stop/Monitor) system services running within the clusterCDAP ServicesThese are mission critical CDAP system services running in YARN containers on a Hadoop cluster.Thelifecycle on these services are managed by the CDAP Server. Below are the services running on the clusterDataset Executor: Responsible for managing dataset lifecycleMetadata: Responsible for managing metadata for applications and datasetsLog and Metrics Aggregator: Responsible for aggregating and indexing logs and metrics across allapplications and datasetsTransactions: Responsible for providing consistency guarantees across applications and datasetsExplore: Responsible for exposing the querying (SQL) interface for datasetsStream: Responsible for data ingestion either in realtime or batchCask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

Use-case(s)Following are a few use-cases that CDAP is well-suited for and are being used within customer environments.Data LakeHigh VolumeStreaming AnalyticsInformationSecurity ReportingReal-time brandand marketingcampaign monitoringCask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

Use-case(s)Following are a few use-cases that CDAP is well-suited for and are being used within customer environments.Data LakeBuilding an enterprise data lake requires building a reliable, repeatable and fullyoperational data management system, which included ingestion, transformations &distribution of data . It must support varied data types and formats, and must be able tocapture data flow in in various ways. The system must support the following:Transform, normalize, harmonize, partition, filter and join dataInterface with anonymization and encryption services external to the clusterGenerate metadata for all data feeds, snapshots and datasets ingested, and make it accessible throughAPIs and webservicesPerform policy enforcement for all ingested and processed data feedsTracking and isolating errors during processingPerforming incremental processing of data being ingestedReprocessing data in case of failures and errorsApply retention policies on ingested and processed datasetsSetup common location format (CLF) for storing staging, compressed, encrypted and processed dataFiltered views over processed datasetsMonitoring, reporting, and alerting based on thresholds for transport and data quality issues experiencedduring ingestion. This helps provide the highest quality of data for analytics needs.Annotate Datasets with business/user metadataSearch Datasets using metadataSearch Datasets based on schema field names and typesManage data provenance (lineage) as data is processed/transformed in the data lakeOutcomeA team of 10 Java (Non-Hadoop) developers were able to build an end-to-end ingestion system with thecapabilities described above using CDAP. Lower barrier to entry.These developers provided a self-service platform to the rest of the organization(s) to ingest, process andcatalog data. Abstractions helped them build at a much faster pace and get it to their customers faster.Time to market.The ingestion platform standardized and created conventions for how data is ingested, transformed andstored on the cluster, allowing the platform users to on-board at much faster rate. Time to value.CDAP’s native support for incremental processing, reprocessing, tracking metadata, workflow, retention,snapshotting, monitoring and reporting expedite the efforts to get a system to their customers. Time tomarket.CDAP is installed in 8 clusters with 100s of nodes.Data Lake users were able to locate Datasets faster and had faster access to metadata, data lineage anddata provenance. This allowed them to efficiently utilize their clusters and also aided them in datagovernance, auditability, and improving the data quality of Datasets. CDAP Tracker provided this set ofcapabilities.Cask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

High Volume Streaming AnalyticsBuilding a high speed, high volume streaming analytics solutions with exactly-oncesemantics is complex, resource intensive and hard to maintain and enhance. Thisuse-case required data collection from web logs, mobile activity logs and CRM data, inreal-time and batch. The data collected was then organized into customer hierarchiesand modeled to deliver targeted ad campaigns and marketing promotion campaigns. Italso had to provide advanced analytics for tracking the campaigns in real-time. Thisapplication had to support the following:Support processing of 38 billion transactions per day in real-timeCategorizing customer activity into buckets and hierarchiesGenerating unique counts in real-time, to understand audience reach, tracking behavior trend, and the like.Generate hourly, daily, monthly and yearly reports on multiple dimensionsProvide unique stat count on an hourly basis rather than weeklyReprocessing data without side effects due to bug fixes and new featuresExactly-once processing semantics for reliable processingProcessing data both in real-time and batch.OutcomeCDAP’s abstraction and its real-time program simplified building this application and in getting it tomarket faster. Time to market.The team replaced a MapReduce based batch-system to a realtime system, delivering insights everyminute instead of days.CDAP’s exactly-once and transactional semantics provided high-degree of data consistency duringfailure scenarios, making it easy to debug and reason the state of data.CDAP’s Standalone and Testing frameworks allowed the developers to build this application efficiently.No distributed components were required to run functional tests.Cask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

Information Security ReportingIn a large enterprise environment there are traditional sources that house a great deal ofdata. There is a constant need to load data into Hadoop clusters to perform complexjoins, filtering, transformations and report generation. Moving data to Hadoop iscost-effective as there is the need to run many complex, ad-hoc queries that wouldotherwise require expensive execution on traditional data storage and queryingtechnologies.The Customer has been attempting to build a reliable, repeatable data pipeline for generating reports acrossall network devices which access resources. Data is currently aggregated into five different Microsoft SQLServers. Aggregated data is then periodically (once-a-day) staged into a secured (Kerberos) Hadoop cluster.Upon loading the data into the staged area, transformations (rename fields, change type of field, project fields)are performed to create new datasets. The data was registered within Hive to run Hive SQL queries for anyad-hoc investigation. Once all the data is in final independent datasets, the next job is kicked off -- that joinsthe data from across all five tables to create a new uber table that provides a 360 degree view for all networkdevices. This table is then used to generate a report that is part of another job. Following are the challengesthe customer faced:Ensuring that the reports aligned to day boundariesRestarting the failed jobs from the last point where they had failed (had to reconfigure pipelines to restartfailed jobs)Adding new sources required a lot of setup and development timeInability to test the pipeline before it was deployed — this lead to inefficient utilization of the cluster as allthe testing was performed on the clusterThey had to cobble together a set of loosely federated technologies -- Sqoop, Oozie, MR, Spark, Hiveand Bash ScriptsOutcomeThe in-house Java developers with limited knowledge of Hadoop built and ran the complex pipelines atscale within two weeks after four (4) hours of trainingThe visual interface enabled the team to build, test, debug, deploy, run and view pipelines duringoperationsThe new process reduced system complexity dramatically, which simplified pipeline managementThe development experience was improved by reducing inappropriate cluster utilizationTransforms were performed in-flight with error record handlingTracking tools made it easy to rerun the process from any point of failureCask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

Real-time brand and marketing campaign monitoringEnterprises use Twitter to know when people are talking about their brand andunderstand sentiment toward their new marketing campaigns. Real-time monitoringcapabilities on Twitter allows them to keep a close eye on the results of marketing efforts.Developing a real-time pipeline that ingests the full Twitter stream, then cleanses,transforms, and performs sentiment and multi-dimensional analysis of the Tweets thatwere related to campaign delivers a valuable real-time decision making platform. The aggregated data isexposed through REST APIs to an internal tool for visualization, making consumption of the output easier.The pipeline is built using Storm, HBase, MySQL and JBoss. Storm is used to ingest and process the streamof Tweets. The Tweets are analyzed using NLP algorithms to determine sentiment. They are aggregated onmultiple dimensions like number of re-tweets and attitude (positive, negative or neutral). The aggregations arestored in HBase. Periodically (twice-a-day) the data from HBase is moved into MySQL. JBoss exposed RESTAPIs for accessing the data in MySQL.The goal of this use-case was to reduce the overall complexity of the pipeline, moving away from maintaininga separate cluster for processing the real-time Twitter stream, integrate NLP scoring algorithms for sentimentanalysis and exposing the aggregated data from HBase with lower latency, thereby reducing the latencybetween the data being available in HBase to delivery via REST API. The result - an easy to build, deploy andmanage real-time pipeline with better operational insights.OutcomeCask Hydrator pipeline for processing full twitter stream was built in 2 weeks.Cleansing, Transforming, Analyzing and Aggregating tweets at about 6K/sec in-flight.Consolidated infrastructure into a single Hadoop cluster.Java Developers were able to build the pipeline and plugin with less learning curve.CDAP Service on OLAP Cube reduced the expensive data movement and reduced the latency betweenthe aggregation being generated to the results being exposed through REST APIs allowing them to makebetter decisions faster.CDAP and Cask Hydrator seamlessly transparency provided easy operational insights through customdashboards and aggregated logs for debugging.Cask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

A day in life with CDAPThis section of the document describes how a day would go for a developer and operations team memberbuilding and deploying an application or a solution on Hadoop in production using CDAP. In order todemonstrate this we will take a common use-case with an organization using Hadoop.Use-caseJoltie, a Java developer, and Root, a operations team member, work in anenterprise. They have been tasked with building and operationalizing adata pipeline for processing data that is ingested and available on HDFS.The data is delivered to a standard directory on a daily basis with anapproximate size of 600GB per day. The processing pipeline has to beoperationalized to process daily data within a SLA of 3 hrs. Developer andOperations only have access to CDAP, not Cask Hydrator.The data pipeline must include the following :1234A dataset integrity job thattakes a pass over all thedata from a day to check ifthe data on HDFS is reliableenough to be processed.A transformation job thatwill process a day's worthof data -- appliestransformations, filteringand field level validation.The output of the job is anew transient dataset.The output from the step 2is then picked up byanother job that interactswith the encryption systemto encrypt certain fields inthe feed. This job willgenerate two outputs -one that is encrypted andthe other that is not. Thejob also writes both theoutput sets to datasetspartitioned by day. Thesedatasets are explorableusing Hive.The in-encrypted partitionoutput is further processedto build and update a datamodel in HBase. The datain HBase is also explorablethrough Hive.Conditions1The jobs in step 1 and step2 can run in parallel.2The job in step 3 is executed only if thejob in step 1 clears that the data isreliable and step 2 job is successful.3The job in step 4 isdependent on successfulcompletion of step 3.EnvironmentThe users are provided a very constrained environment for developing and testing the pipeline. In total thereare three (3) clusters made available to them:A Dev Cluster - 2% capacity of production clusterA QA Cluster - 5% capacity of production clusterA Production ClusterCask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

For Joltie - The DeveloperJoltiea mid-level Java developer with experience building web applications isvery excited about this opportunity to build a Big Data application. Shehas taken the initiative to learn about the basics of Hadoop. She isintimidated by the complexity of Hadoop and has been looking atdifferent examples of how to process data on Hadoop. Following areadditional requirements that she has been tasked with:Make the application code modular and testableCommit changes on regular basisWrite unit test cases for each piece functionality that is added to the applicationSet up a job in Jenkins that runs unit tests on every check-inSetup an end-to-end test for the application on the QA clusterInstrument the application correctly to provide more insights into business processingThe application should handle some critical edge casesIn case of an error in the pipeline, the user should be able to fix the issue and easily reprocessThe pipeline must support incremental processingThe pipeline must handle the case where data arrives lateJoltie has to work with Root (operations team member) to successfully deploy the applicationto QA and productionJoltie has to use CDAP to build this applicationMaterial AvailableCDAP DocumentationTools NeededCDAP SDKCDAP Apps & ExamplesDesign & ResearchMavenJoltie starts by looking at the pipeline requirements and in parallelIDEspends time reading CDAP documentation and running examples.LaptopShe is delighted that she doesn’t need to install Hadoop.NodejsJavaCask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

DevelopmentDAY1First Cut of the ApplicationJoltie starts up by creating a base plate CDAP Application using CDAP Maven Archetype.All the necessary dependencies are included, so she is ready to get startedShe then modifies the Application to build a Directed Acyclic Graph (DAG) for processing input data, asdescribed in the use-case, using the Workflow API provided by CDAP.She also uses the CDAP JUnit scaffolding to build a unit test for testing the Workflow. She finds it’s veryuseful to have the ability to test the workflows before being deployed on a cluster or anywhere else.Joltie builds the project using Maven to generate the application artifact.She then starts a CDAP Standalone version on her laptopShe deploys the artifact into CDAP Standalone and plays around with it. She cannot test it at scale, butis able to experience working with Application as it would have been in the cluster.She Iterates through the development cycle without having to touch the a clusterShe goes home feeling like he has accomplished a great deal and has developed the first version of theApplication by herself.DAY2Application Enhancement and CI SetupIt’s a beautiful day. Joltie is ready to include a few of the edge cases that were provided as requirementsand take the application to completion.She reads about Partitioned Fileset Datasets and configures her Application to include this in orderto solve the edge cases. The Partitioned Fileset Datasets are transactional and provide consistencyrequired to handle new partitions and also handle errors efficiently.She iterates on it a few times to make sure she includes test scenarios for edge cases and also tries thesame within CDAP Standalone. she is able to simulate a few of the scenarios, but she has to try this on areal cluster to make sure it works as expected under error conditions.Next, she is ready to setup a Jenkins job to periodically trigger tests on check-in. she specifies theconfigurations and commands to run the tests the same way she would in a shell.She now wants to setup an end-to-end test of her application -- she is not sure how she can accomplishthis, so, she opens an issue within Cask’s support portal. She gets a response back within a few hourswith information around how she can set-up the end-to-end test.She follows the instructions specified by a Cask representative to set-up an end-to-end job on Jenkins.She has accomplished a lot on the second day!Cask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

DAY3Operations HandoffJoltie reaches office early, with a spring in her step, to get his Application into the QA environment.Before starting to work with Root, Joltie goes in and adds metrics to track the many business metricsthat would be useful to debug issues and provide insights into the Application itself. She uses theCDAP-provided Metrics APIsShe meets Root and explains the Application and what it does to him. Root, having worked on Hadooppreviously, is not very excited about a new Application being dropped on his lap for operations.Joltie starts of by deploying the Application into a CDAP Standalone and starts it. He proceeds to explainthe different aspects of Application/Workflow and how the data will be processed to Root. Root hasalready started feeling comfortable and is asking a lot of questions around how he can monitor thisApplication.Joltie then easily builds a dashboard using the CDAP Console and highlights the metrics that Root canuse to monitor the application.Cask Data Application Platform - CDAP 101Copyright 2016 to Cask Data Inc. Proprietary and Confidential

For Root - The Operations GuyRootthe operations guy is charted with deploying and managingthe data pipeline in QA and production environments. Inorder to be successful with this project, Root needs toaccomplish the following:Install and configure CDAP on a cluster using the package manager provided by the distribution he is usingSecure the CDAP instance with authentication, so that only authenticated users are allowed to gain accessto the instanceEnsure that the authentication can be integr

JDBC / ODBC Driver The CDAP JDBC and ODBC drivers enable users to . (BI) applications with JDBC or ODBC support. The driver achieves this integration by translating Open Database Connectivity (JDBC/ODBC) calls from the application into SQL and passing the SQL queries to . Sensu, Cacti and Sp