Using Apache Spark, Apache Kafka And Apache Cassandra

Using Apache Spark,Apache Kafkaand Apache Cassandrato Power Intelligent Applications

Apache Cassandra is well known as the database of choicefor powering the most scalable, reliable architecturesavailable. Apache Spark is the state-of-the-art advanced andscalable analytics engine. Apache Kafka is the leading streamprocessing engine for scale and reliability.Deployed together, these technologies give developers thebuilding blocks needed to build reliable, scalable andintelligent applications that adapt based on the data theycollect.This paper discusses the use cases, architectural pattern andoperations considerations for deploying these technologiesto deliver intelligent applications.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 02

Use CasesInternet of ThingsAt the core of an IoT application there is a stream of regularobservations from (potentially) a large number of devices oritems with embedded electronics (e.g. switches, sensors, tags).A stream of IoT data is just “big data”, but analysing that bigdata in a way that drives actions, recommendations, orprovides information is where the application delivers value.Apache Cassandra is extremely well suited to receiving andstoring streams of data. It’s always-on availability matches theconstant stream of data sent by devices to ensure yourapplication is always able to store data. In addition, its nativestorage formats are well suited to efficient storing and usingtime series data such as that produced by IoT devices. Thescalability of Apache Cassandra means you can be assured thatyour datastore will smoothly scale as the number of devicesand stream of data grows.The powerful analytics capabilities and distributed architectureof Apache Spark is the perfect engine to help you make senseand make decisions based on the data you’re receiving fromyour IoT devices. Spark’s stream processing can quicklydetermine answers from short-term views of your data as it’sreceived. For analysis running over longer time periods, theSpark Cassandra connector enables Spark to efficiently accessdata stored in Cassandra to perform analysis.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 03

In this context, Apache Kafka is often used as a reliablemessage buffer. In many IOT scenarios, the flow of data fromdevices is constant and the devices have very limitedcapacity to buffer data in the event the central processingservice is unavailable. Events from the devices can be writtento Kafka when first received and then picked up andprocessed by the downstream applications. This ensuresevents are not lost even if the processing elements for thecentral system become backed up or suffer downtime. Inaddition, use of Kafka in this manner easily allows additionalconsumers of the event stream to be added to the system.For example, your initial implementation may have a simpleapplication that just saves data to Cassandra for later use butyou then you add a second application that performs realtime processing on the event stream. Kafka Streams may alsobe used as an alternative to Spark Streaming for real timestream processing.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 04

Financial ServicesThe pressures for financial services companies to gaina technological edge in data processing are coming not onlyfrom the competition but also from consumers. Gaining acompetitive edge requires systems that can collect andquickly analyse vast streams of data. Consumers expect thatthe systems they interact with will be instantly up to date,always available and, increasingly, be aware of the context ofall their previous interactions and related information.Addressing these joint pressures, while containing technologycosts, requires the adoption of new generation architecturalpatterns and technologies. Apache Cassandra, Apache Kafkaand Apache Spark are technologies that are ideally placed toform the core of such an architecture. The applicability ofthese technologies in financial services has been proven manytimes by leading organisations such as ING and UBS.One common application we see for Cassandra in financialservices is as a persistent cache to support high volume clientrequests. In particular, we see this requirement with banksimplementing the Payment Services Directive (PSD2) in theEU. This leverages Cassandra’s extreme reliability andbuilt-for-the-cloud architecture to enable financial servicesorganisations to deliver always-on service and avoid the highcost of scaling their legacy (often mainframe) architectures tomeet increased client interactions needs. Spark is oftenincluded in this architecture to enrich the cached data withsophisticated analysis of trends and patterns in the dataenabling user-facing applications to make this analysis withinteractive response times. Kafka often sits in this picture as amessage bus to connect the core processing system tomultiple downstream consumers.USING APACHE SPARK AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 05

OthersThe two use-cases above are great examples where we see regularadoption of Spark, Kafka and Cassandra. However, there are manyother business problems where the three technologies can combine toprovide an ideal solution. Some examples that we have seen include:Ad-TechRelying on the low-latency (low double digit ms)responsiveness and always-on availability of Cassandra tomake online advertising placement decisions backed by deepanalysis calculated with Spark. Massive flows of inboundevents and information can be managed with Kafka.Application MonitoringWe use a combination of Spark and Cassandra in our ownmonitoring system that monitors close to 1500 servers.Cassandra seamlessly handles a steady stream of writes withmetrics data while Spark is used to calculate regular roll-upsto allow viewing summarised data over long time periods.Kafka acts as a centralisation point for the messages and alsoa message buffer.Inventory Management, particularly in travelUse Cassandra to track inventory records and Spark to analyseavailable inventory to determine dynamic pricing, capacitytrends, etc.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 06

Architectural PatternsBatch UpdatingAt a more rudimentary level, many Cassandra applications have aneed for periodic batch processing for data maintenance. Whilethis can include summarisation it can also include requirementslike implementing complex data expiry rules. Running thesebatches through a single threaded (or single machine) batchengine will not scale to the same extent your Cassandra clusterwill. Implementing these batch jobs in Spark not only provides apre-built set of libraries to assist with development of the dataprocessing functionality but also the frameworks to automaticallyscale the jobs and scale and execute processing logic on the sameservers where the data is stored.Stream EnrichmentFor most applications, a strong design will store in a singleCassandra table all of the information required to service aparticular read request (i.e. the data will be highly denormalised). Insome cases this denormalisation process will require calculating orlooking up additional data to add to a stream before the stream ofdata is saved. Using Spark Streaming to process data before savingto Cassandra provides a scalable and reliable technology base toimplement this pattern. Kafka Streams is an alternative engine forimplementing this form of stream enrichment.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 07

Lambda ArchitectureThe Lambda Architecture is an increasingly popular architectural pattern forhandling massive quantities of data through both a combination of stream andbatch processing. With the Lambda Architecture, you maintain a short-term,volatile view of your data (the speed layer) and a longer term, morepermanent view (the batch layer) with a service to join views across the two(the serving layer).With Spark and Cassandra, you have the key architectural building blocks youneed to implement the Lambda Architecture. Spark Streaming is an idealengine for implementing the speed layer of the architecture (potentially withresults stored in TTL’d tables in Cassandra) while Spark can also be used toperform the longer-term batch calculations and store results in Cassandra.Kappa ArchitectureThe Kappa Architecture takes the next step from the Lambda Architecture,removing the batch layer and treating the stream of events as the immutablerecord of system state. Stream processing maintain summary views as thestream is processed. If the logic of summary views needs to change then thestream processing logic is updated and the saved streams reprocessed. TheKappa Architecture removes the need to maintain separate stream and batchlogic that is required for the Lambda Architecture.Once again, the combination of Spark and Cassandra gives you thearchitectural components you need to implement Kappa Architecture. SparkStreaming is an ideal processing engine to undertake the calculations neededon the stream of data. Apache Cassandra can be used both as the long term,immutable store of the data stream and as a store for the results of the streamcalculations that are used by the serving layer. An alternative is to use ApacheKafka as your immutable event store and Apache Cassandra as the store forthe materialized views calculated based on these events.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 08

OperationsOperating as part of a mission-critical application isthe normal mode of operation for Cassandra andthere is a well established body of knowledge abouthow to operate Cassandra to achieve the highestlevels of availability. Although Kafka is a little newerit is also widely operated at the highest levels of scaleand reliability.Spark, on the other hand, is often run to provide ananalytics environment for use by a small number ofdata scientists. In this situation, reliability andpredictable performance are not as critical as whenSpark is deployed as a component of a productionapplication. This section of the paper describes someof the considerations to be applied when deployingSpark for production usage.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 09

Management EnvironmentThe key to reliable operations of any technology is to have a solidoverall management environment including aspects such as:Automated (or at least well controlled)deployment and configuration management.High quality testing of new configurations priorto deployment.Backup and disaster recovery procedures.Appropriate monitoring, and systems and peoplethat are paying attention to what is beingreported by that monitoring.Rigorous incident response procedures andwell-trained staff.None these items is specific to Kafka, Spark or Cassandra. However,introducing production usage of these technologies will requireexamination of each of these areas to ensure they are fit forpurpose with introduction of new architectural components andapplications.USING APACHE SPARK, APACHE KAFKA AND APACHE CASSANDRA TO POWER INTELLIGENT APPLICATIONS 10