A Simple Framework For Building Predictive Models

3.1. Use Cases4. Modelling FrameworkThe table below lists predictive analytics businessapplications. The first column names the type ofbusiness benefit, and the second column identifies thetype of customer prediction required – that is, whichbehavior or action must be predicted to undertakeeach business application. As there are many suchapplications, this list includes only the most pervasivein commercial deployment to date.“The best way to predict the future is to create it.”BusinessApplicationWhat isPredictedReference orCompanyCase StudyCustomer retentionCustomerdefection/churn/attritionReed Elsevier,and Telenor,Published articleeCommerceHow to sellsuccessfully onlineAmazon.comDirect marketingCustomer responseCharles SchwabPublished tflix Prizeleader, HSBC &Amazon.comBehavior-basedadvertisingGoogle, Yahoo!Which ad customer and ClickwillForensics,click“ 1 million”case studyEmail targetingMessage that willgenerate customerresponseCredit scoringDebtor risk or fraud Wells FargoFundraising fornonprofitsDonation amountInsurance pricingand selectionInsuranceApplicant response,Journal, Pinnacolinsured riskAssuranceWorld WildlifeFundNRAThere are many more applications of predictive analytics,including collections, supply chain optimization, humanresource decision support for recruitment and humancapital retention, and market research survey analysis.The way predictive scores help your business dependson the customer behavior predicted – just aim predictiveanalytics towards the right customer predictiongoal and fire away. This is why it is sometimes said,“If you predict it, you own it!”4.1 Recommended Predictive ModellingBusiness RequirementsWhile they may share many characteristics andtechniques, building statistical or predictive modelsfor a scientific or university research context can bedifferent than a similar exercise in a business context.For business-oriented projects, here are some basiccore modelling requirements and considerations. Formulate a clear and transparent framework:- What are the objectives? What are wemodelling for?- Predictive models alone do not createbusiness value, but rather need to beeffectively deployed either into a decision making or business process. Make room for using business knowledge in theprocess:- Involve business users in the modellingprocess to ensure clear requirements,communication and purpose verification.- Business experience and heuristics maycontain valuable nuggets for how to use thedata.- Do solutions pass the “Does it makebusiness sense” test? Make sure that data are appropriate formodelling:- Quality, scope and quantity suitable to meetobjectives- Structured in line with underlying processesbeing modeled to maximize the signal Don’t just make best statistical fits to historicaldata:- Test predictive power and consistency ofpatterns over time and across validationsamples Think about managing the model:- How to integrate the model into existingbusiness systems- After implementing, how to track andmanage the interactions, and how tomaintain, tweak, or update the modelgoing forward in an efficient mannerA Simple Framework for Building Predictive Models 6

4.2 The Modelling ProcessScope“The devil is in the details.”A clear specific objective for the model is required.Each model is developed for its own specific purposeand cannot be used effectively in another situation.For example a model that predicts ecommerce onlinecustomer churn cannot be used to predict credit cardchurn. An example of a clearly defined model objectivecontains the event or action that the model is to predictand the period when it is likely to happen. For example,the objective could be to predict customers that arelikely to miss a credit card payment within the nextmonth.Let’s take a look at the process of predictive modelling.There is a wealth of information on the Web about this.However, to understand the strategic areas, it is usefulto break down the process of prediction into just a fewessential components.The below diagram is a good depiction of the predictivemodelling process across a wide range of businesses.Prepare the DatasetThe dataset created to use for the model might broadlycover diverse information, such as product, case,behavioral, demographic, geographic, competitordetails or weather. The variables that are not included inthe dataset will not form part of the prediction. Variablescover both static fields such as income and triggerssuch as change in spend. Both technical and businesspeople need to be involved in the decisions relatingto the contents of the dataset. Focus needs to be onthe behavioral information as this is more powerful forpredictions than demographic data. It is useful to giveconsideration to which customers to exclude from themodel build process.Broadly speaking, predictive analysis and modelling canbe divided into three parts: Plan, Build, and Implement.4.2.1 Plan the ModelThe next step includes scoping and planning. In thePlanning section, the sub-phase activities here arecalled Scope and Prepare. This part of the exercisemight take around 40% of the total time.To build a predictive model, you first need to assemblethe datasets that will be used for training. You mustformulate clear objectives, cleanse and organize thedata, perform data treatment including missing valuesand outlier fixing, make a descriptive analysis of thedata with statistical distributions, and create data setsused for the model-building.Customers need to be excluded if they are goingto impair the performance of the model. Potentialexclusions include things like bad debt, staff and newcustomers as they have insufficient history. The actualexclusions applied relates to the specific purpose of themodel. For example, if the model is to predict customersthat are likely to carry bad debts, bad debt customerswould be included in the model. In practice, exclusionsare applied as filters when creating the dataset andshould be noted down.4.2.2 Build the ModelHere you will write model code, build the model,calculate scores, and validate the data. In the Buildsection, the Sub-phases are known as Model andValidate. This might take around 20% of the overall time.This is the part that can be left with the data scientistsor technical analysts. The technical aspects of buildingmodels are not covered in this paper.A Simple Framework for Building Predictive Models 7

The model will be built using a sample from the data setcreated. The resulting model will contain a subset of theoriginal list of variables considered for the model. This isacceptable and happens because some of the variablesconsidered for the model will be correlated with eachother, for example, a product type and product family,or the floor number and height of building. Othervariables will have been discarded as they add little ornothing to the model’s predictive power.Deploy the ModelIt’s time to apply the model. The model will be builton a subset of data. Once the model is complete andhas been validated, it will be run over the customer,product, or case base.AssessThe model developed will be an equation that, whenapplied to the customer base, will allocate a scoreto each customer. The score represents a customer’slikelihood ‘to do’ whatever the model is predicting,such as predict a customer’s expected order value orlikelihood to churn within a month. If you are buildinga model for product failures, you may have a score thatcalculates the likelihood of a product to fail in a giventime frame.It is typical to generate rankings from your model– depending what you are modelling – customersor products. You will want to understand theperformance of your model. The ranking scores allowa customer or product base to be ranked in order ofthe predicted score, such as from most likely to leastlikely to churn or from most popular product to least. Inreality, some customers will churn and some customerswill not. Therefore, in absolute terms, these predictionswill not be accurate. The ranked list provides a superbbase upon which to vary the treatment, and thereforethe level of service or marketing spend, to groups ofcustomers.Validate the ModelMonitorCalculate a ScoreTypically models are validated against a hold out group.This group contains customers that have not beenincluded in the development of the model. As such,they represent a group of previously unseencustomers that are representative of the customerbase. To achieve an accurate prediction of lift, the holdout group must not be made up of the customersthat have been excluded from the modeldevelopment process. The same idea applies ifyou are building a product failure score model.4.2.3 Implement the ModelHere you will deploy and apply the model, rankcustomers or products, use the model outcomes forsome business purpose, estimate model performance,assess and monitor the model, and drive initiativesbased on the model. The Sub-phases here are knownas Deploy, Assess, and Monitor.You need to think about accessing, storing, and usingthe data. This might take around 40% of the overalltime, require IT department work, and can be anongoing operational exercise when the model is usedcontinuously for the business.You want to consider how often to run the model acrossthe base and generate scores. You will want to monitorthe performance of the model on an ongoing basis, andperhaps take advantage of new data or techniques asthey become available.4.3 Modelling Longevity and Considerations“The simplest solution is the best.”It is useful to think of the deployment of predictivemodels in a continuous improvement scenario. Youdon’t just build the model and leave it alone. The modelmay take nourishment, watering, monitoring, and careand can improve over time as conditions change andnew data come in.A model does not have to be extremely complicatedin order to make good predictions. Like in coding,simplicity is a good guideline. It is useful to keepOccam’s Razor in mind when considering how to build amodel: Some people say Occam’s Razor is “the simplestsolution is the best.” However, if you look at the originaltext, it says “Do not multiply entities beyond necessity.”Generally, the analyst may make assumptions whenbuilding a model. However, nature does not assumeanything before forcing an event to occur. The fewerassumptions there are in a predictive model, the greaterA Simple Framework for Building Predictive Models 8

will be the predictive power. Clearly, attributes that arenot in the model will have no effect on the model’spredictions.The speed of changing business conditions is a factor inhow well the model will perform over time. For example,information technology is a faster-moving industry thaninsurance. Once a model is in use and driving actions,there is a requirement for tracking and managing theinteractions. It is a good idea to refresh model scoresover time, sometimes frequently depending on theindustry.Control files will be required to test initiatives and totest the model performance. The results from this willhelp determine how frequently the model needs tobe refreshed. As a rule of thumb, a model needs to bereviewed and possibly rebuilt annually.4.3.1 Types of Predictive Models“There are three types of lies -- lies, damn lies, andstatistics.”Many types of predictive models have been developedover the years that are useful for different classes ofproblems. Applying these techniques is the domain ofstatisticians and data scientists, and is intentionally notcovered in this short paper.1. Business RulesA business rule is a rule that defines or constrains someaspect of business and always resolves to either trueor false. Business rules are intended to assert businessstructure or to control or influence the behavior of thebusiness.2. Classification and Decision TreesA decision tree is a decision support tool that uses atree-like graph or model of decisions and their possibleconsequences, including chance event outcomes,resource costs, and utility. It is one way to display analgorithm.3. Naive BayesIn machine learning, naive Bayes classifiers are a familyof simple probabilistic classifiers based on applyingBayes’ theorem with strong (naive) independenceassumptions between the features. The techniqueconstructs classifiers: models that assign class labels toproblem instances, represented as vectors of featurevalues, where the class labels are drawn from somefinite set.4. Linear RegressionIn statistics, regression analysis is a statistical processfor estimating the relationships among variables. Linearregression is an approach for modelling the relationshipbetween a scalar dependent variable Y and one ormore explanatory variables (or independent variables),denoted X. The case of one explanatory variable iscalled simple linear regression. More than one variableis called multivariate.5. Logistic RegressionIn statistics, logistic regression, or logit regression, orlogit model is a regression model where the dependentvariable (DV) is categorical or binary.6. Neural Networks (NNs)An Artificial Neural Network (ANN) is an informationprocessing paradigm that is inspired by the waybiological nervous systems, such as the brain, processinformation.7. Machine LearningMachine learning is a type of artificial intelligence (AI)that provides computers with the ability to learn withoutbeing explicitly programmed. Machine learning focuseson the development of computer programs that canteach themselves to grow and change when exposedto new data.8. Support Vector Machines (SVMs)In Machine Learning, an SVM is a supervised learningmodel with associated learning algorithms that analyzedata used for classification and regression analysis. AnSVM maps input data vectors into a higher dimensionalspace, where an “optimal hyperplane” that separatesthe data is constructed. An SVM is a discriminativeclassifier formally defined by a separating hyperplane.In other words, given labeled training data (supervisedlearning), the algorithm outputs an optimal hyperplanewhich categorizes new examples.9. Natural Language Processing (NLP)Since NLP (content analytics) feeds predictive andprescriptive analytics, it is included here. NLP factA Simple Framework for Building Predictive Models 9

extraction can be used with descriptive statisticalformulae to make use of the wealth of unstructureddata.4.5 Practical Considerations When Working withthe Data4.4 Supervised and Unsupervised Learning1) Asking the Right Questions“Statistics can be made to prove anything - even thetruth.”“Correlation does not imply causality.”What are the business objectives of the project? Does itmake business sense?SVMs, decision trees, NNs and regression models usesupervised learning to create the mapping functionbetween a set of input data fields and a target variable.The known outcome is then used as a teacher whosupervises the learning of her pupil. Whenever the pupilmakes a mistake, the teacher provides her with the rightanswer in the hopes that the pupil will eventually get itright. For instance, when presented with a specific set ofinputs, her output will match the target.Awash in reams of data, it is critical that companiesask the right questions. Being specific is important.Predictive analytics is most efficient when used todetermine the answer to a narrow inquiry, such as thelikelihood of customer A to buy product X at time Y forprice Z, rather than the likelihood of customers buyingproduct X.Unsupervised learning requires no teacher or target.Clustering techniques fall into this category. Data pointsare simply grouped together based on their similarity. Inan ecommerce website analysis, online shoppers mightbe grouped into window shoppers or power buyers. Incase of customer churn, a clustering technique couldpotentially assign different clusters to churners andnon-churners even though the outcome is not availableduring model training.It is one thing to have someone say, “Build me apredictive model to mint money.” Is that realistic? If thegoal looks realistic, one must find out if there are evendata available to do something reasonable. Sometimesthe data will not be accessible, or it will not be orderedor structured well enough.Black-box is a term used to identify certain predictivemodelling techniques that are not capable of explainingtheir reasoning because you can’t really know whatis happening in the Black-box. Although extremelypowerful, techniques such as NNs and SVMs fall intothis category. Very often, an explanation or reason forthe model decision is required; for example, when a riskscore is used to decline a loan application or a creditcard transaction.Whenever explaining is a must, you need to considerusing a predictive modelling technique that clearlydescribes the reasons for its decisions. Scorecards fitsuch criteria very well. Based on regression models,scorecards are a popular technique used by financialinstitutions to assess risk. With scorecards, all data fieldsin an input record are associated with specific reasoncodes. Thus, it is possible to explain to the customerwhy a given decision was rendered.2) Identify Available Data3) Data Accuracy and CleanupData scientists must be aware that not all data isaccurate, arrive at an estimate of bad data, and correctfor it in their studies. Data can be bad for any numberof reasons, including self-reporting errors, corruptedfiles, poorly phrased questions, incomplete dataaggregation, and poor standardization methods.It is critical that data scientists quickly recognize andfilter bad data f

predictive analytics and predictive models. Predictive analytics encompasses a variety of statistical techniques from predictive modelling, machine learning, and data mining that analyze current and historical facts to make predictions about future or otherwise unknown events. When most lay people discuss predictive analytics, they are usually .