Technology Assisted Review (Tar) Guidelines

TECHNOLOGY ASSISTED REVIEW (TAR) GUIDELINESEDRM/DUKECHAPTER ONEDEFINING TECHNOLOGY ASSISTED REVIEWCHAPTER ONEDEFINING TECHNOLOGY ASSISTED REVIEWA. INTRODUCTION . 1B. THE TAR PROCESS . .21.2.3.4.5.ASSEMBLING THE TAR TEAM .COLLECTION AND ANALYSIS “TRAINING” THE COMPUTER USING SOFTWARE TO PREDICT RELEVANCY .QUALITY CONTROL AND TESTING .TRAINING COMPLETION AND VALIDATION . . .22333A. INTRODUCTIONTechnology assisted review (referred to as “TAR,” and also called predictivecoding, computer assisted review, or supervised machine learning) is a review processin which humans work with software (“computer”) to train it to identify relevantdocuments. 2 The process consists of several steps, including collection and analysisof documents, training the computer using software, quality control and testing, andvalidation. It is an alternative to the manual review of all documents in a collection.Although there are different TAR software, all allow for iterative andinteractive review. A human reviewer 3 reviews and codes (or tags) documents as“relevant” or “nonrelevant” and feeds this information to the software, which takesthat human input and uses it to draw inferences about unreviewed documents. Thesoftware categorizes documents in the collection as relevant or nonrelevant, or ranksthem in order of likely relevance. In either case, the number of documents reviewedmanually by humans can be substantially limited while still identifying thedocuments likely to be relevant, depending on the circumstances.In fact, the computer classification can be broader than “relevancy,” and can include discoveryrelevance, privilege, and other designated issues. For convenience purposes, “relevant” as used in thispaper refers to documents that are of interest and pertinent to an information or search need.3 A human reviewer is part of a TAR team. A human reviewer can be an attorney or a non-attorneyworking at the direction of attorneys. They review documents that are used to teach the software. Weuse the term to help keep distinct the review humans conduct versus that of the TAR software.21

B. THE TAR PROCESSThe phrase “technology assisted review” can imply a broader meaning thattheoretically could encompass a variety of nonpredictive coding techniques andmethods, including clustering and other “unsupervised” 4 machine learningtechniques. And, in fact, this broader use of the TAR term has been made in industryliterature, which has added confusion about the function of TAR, defined as a process.In addition, the variety of software, each with unique terminology and techniques,has added to the confusion by the bench and bar in how each of these software works.Parties, the court, and the service provider community have been talking past eachother on this topic because there has been no common starting point to have thediscussion.These guidelines are that starting point. As these guidelines make clear, allTAR software share the same essential workflow components; it is just that there arevariations in the software processes that need to be understood. What follows is ageneral description of the fundamental steps involved in TAR. 51. ASSEMBLING THE TAR TEAMA team should be selected to finalize and engage in TAR. Members of this teammay include: service provider; software provider; workflow expert; case manager; leadattorneys; and human reviewers. Chapter Two contains details on the roles andresponsibilities of these members.2. COLLECTION AND ANALYSISTAR starts with the team identifying the universe of electronic documents tobe reviewed. A member of the team inputs documents into the software to build ananalytical index. During the indexing process, the software’s algorithms 6 analyzeeach document’s text. Although various algorithms work slightly differently, mostanalyze the relationship between words, phrases, and characters, the frequency andpattern of terms, or other features and characteristics in a document. The softwareuses this features-and-characteristics analysis to form a conceptual representation ofthe content of each document, which allows the software to compare documents toone another.4 Unsupervised means that the computer does not use human coding or instructions to categorize thedocuments as relevant or nonrelevant.5 Chapter Two describes each step in greater detail.6 All TAR software has algorithms. These algorithms are created by the software makers. TAR teamsgenerally cannot and do not modify the feature extraction algorithms.2

3. “TRAINING” THE COMPUTER USING SOFTWARE TO PREDICT RELEVANCYThe next step is for human reviewers with knowledge of the issues, facts, andcircumstances of the case to code or tag documents as relevant or nonrelevant. Thefirst documents to be coded may be selected from the overall collection of documentsthrough searches, identification through client interviews, creation of one or more“synthetic documents” based on language contained, for example, in documentrequests or the pleadings, or the documents might be randomly selected from theoverall collection. In addition, after the initial-training-documents are analyzed, theTAR software itself may begin selecting documents that it identifies as: (i) mosthelpful to refine its classifications; or (ii) most relevant, based on the humanreviewer’s feedback.From the human reviewer’s relevancy choices, the computer learns thereviewer’s preferences. Specifically, the software learns which combinations of termsor other features tend to occur in relevant documents and which tend to occur innonrelevant documents. The software develops a model that it uses to predict andapply relevance determinations to unreviewed documents in the overall collection.4. QUALITY CONTROL AND TESTINGQuality control and testing are essential parts of TAR, which ensure theaccuracy of decisions made by a human reviewer and by the software. TAR teamshave relied on different methods to provide quality control and testing. One popularmethod is to identify a significant number of relevant documents from the outset andthen test the results of the software against those documents. Other software test theeffectiveness of the computer’s categorization and ranking by measuring how manyindividual documents have had their computer-coded categories “overturned” by ahuman reviewer. Yet other methods involve testing random samples from the set ofunreviewed documents to determine how many relevant documents remain. Methodsfor quality control and testing continue to emerge and are discussed more fully inChapter Two.5. TRAINING COMPLETION AND VALIDATIONNo matter what software is used, the goal of TAR is to effectively categorize orrank documents both quickly and efficiently, i.e., to find a reasonable number ofrelevant documents while keeping the number of nonrelevant documents to bereviewed by a human as low as possible. The heart of any TAR process is to categorizeor rank documents from most to least likely to be relevant. Training completion isthe point at which the team has identified a reasonable amount of relevant documentsproportional to the needs of the case.3

How the team determines that training is complete varies depending upon thesoftware, the number of documents reviewed, and the results targeted to be achievedafter a cost benefit analysis. Under the training process in software commonlymarketed as TAR 1.0, 7 the software is trained based upon a review and coding of asubset of relevant and nonrelevant documents, with a resulting predictive model thatis applied to all nonreviewed documents. Here, the goal is not to have humans reviewall predicted relevant documents during the TAR process, but instead to review asmaller proportion of the document set that is most likely to help the software bereasonably accurate in predicting relevancy on the entire TAR set. The softwareselects training documents either randomly or actively (i.e., it selects the documentsit is uncertain about for relevancy that it “thinks” will help it learn the fastest),resulting in the predictive model being updated after each round of training. Thetraining continues until the predictive model is reasonably accurate in identifyingrelevant and nonrelevant documents. At this point, all documents have relevancyrankings, and a “cut-off” point is identified in the TAR set, with documents ranked ator above the cut-off point identified as the predicted relevant set, and documentsbelow the cut-off point as the nonrelevant set.In many TAR 1.0 processes, the decision whether the predictive model isreasonably accurate is often measured based on the use of a control set, which is arandom sample taken from the entire TAR set, typically at the beginning of training,and is designed to be representative of the entire TAR set. The control set is reviewedfor relevancy by a human reviewer and, as training progresses, the computer’sclassifications of relevance of the control set documents are compared against thehuman reviewer’s classifications. When training no longer substantially improvesthe computer’s classifications of the control set documents, training is viewed ashaving reached completion. At that point, the predictive model’s relevancy decisionsare applied to the unreviewed documents in the TAR set. Under TAR 1.0, theparameters of a search can be set to target a particular recall rate. It is important tonote, however, that this rate will be achieved regardless of whether the system is welltrained. If the system is undertrained, an unnecessarily large number of nonrelevantdocuments will be reviewed to reach the desired recall, but it will be reached. Ceasingtraining at the optimal point is not an issue of defensibility (achieving high recall),but rather a matter of reasonableness, minimizing cost of reviewing many extranonrelevant documents included in the predictive relevant set. 87 It is important to note that the terms TAR 1.0 and 2.0 can be seen as marketing terms with variousmeanings. They may not truly reflect the particular processes used by the software, and manysoftware use different processes. Rather than relying on the term to understand a particular TARworkflow, it is more useful and efficient to understand the underlying processes, and in particular,how training documents are selected, and how training completion is determined.8 In many TAR 1.0 workflows, this point of reaching optimal results has been known as reaching“stability.” It is a measurement that reflects whether the software was undertrained at a given pointduring the training process. The term “stability” has multiple meanings. The term “optimum results”is used throughout to eliminate potential confusion.4

Compare this process with software commonly marketed as TAR 2.0. Here,the human review and software training process are melded together; review andtraining occur simultaneously. From the outset, the software continuously analyzesthe entire document collection and ranks the population based on relevancy. Humancoding decisions are submitted to the software, the software re-ranks the documents,and then presents back to the human additional documents for review that it predictsas most likely relevant. This process continues until the TAR team determines thatthe predictive model is reasonably accurate in identifying relevant and nonrelevantdocuments, and that the team has identified a reasonable number of relevantdocuments for production. There are at least three indicators of when completenesshas been reached. The first is when a reasonable recall rate is reached (the humanreview team has reviewed a set of documents that reached a certain level of recallrate, which is calculated/tracked by the TAR software or by a TAR team memberduring the review). The second is the point at which the software appears to beoffering up for review only nonrelevant or a low number of marginally relevantdocuments. The third is the point at which the human review team has identified anexpected, pre-calculated number of relevant documents. In other words, the teamtook a sample before review started to estimate the number of relevant documents inthe TAR set, and then the human team reviewed documents until it reachedapproximately that number. When training is complete, the human reviewers willhave reviewed all the documents that the software predicted as relevant up to thatpoint of the review. If the system is undertrained, then the human reviewers will nothave reviewed a reasonable number of relevant documents for production, and theprocess should continue until that point is reached.Before the advent of TAR, producing parties rarely provided statisticalestimates as evidence to support the effectiveness of their document reviews andproductions. Only on a showing that the response was inadequate did the receivingparty have an opportunity to question whether the producing party fulfilled itsdiscovery obligations to conduct a reasonable inquiry.But when TAR was first introduced to the legal community, parties providedstatistical evidence supporting the TAR results, primarily to give the bench and barcomfort that the use of the new technology was reasonable. As the bench and barbecome more familiar with TAR and the science behind it, the need to substantiateTAR’s legitimacy in every case should be diminished. 9Nonetheless, because the development of TAR protocols and the case law onthe topic is evolving, statistical estimates to validate review continue to be discussed.Accordingly, it is important to understand the commonly cited statistical metrics andThe Federal Rules of Civil Procedure do not specifically require parties to use statistical estimates tosatisfy any discovery obligations.95

related terminology. At a high level, statistical estimates are generated to help thebench and bar answer the following questions: How many documents are in the TAR set?What percentage of documents in the TAR set are estimated to be relevant,how many are estimated to be nonrelevant, and how confident is the TAR teamin those estimates?How many estimated relevant documents did the team identify out of all theestimated relevant documents that exist in the review set, and how confidentis the team in that estimate?How did the team know that the computer’s training was complete?TAR typically ends with validation to determine its effectiveness. Ultimately,the validation of TAR is based on reasonableness and on proportionalityconsiderations: How much could the result be improved by further review and at whatcost? To that end, what is the value of the relevant information that may be foundby further review versus the additional review effort required to find thatinformation?There is no standard measurement to validate the results of TAR (or any otherreview process). One common measure is “recall,” which measures the proportion oftruly relevant documents that have been identified by TAR. However, while recall isa typical validation measure, it is not without limitations and depends on severalfactors, including consistency in coding and the prevalence of relevant documents.“Precision” measures the percentage of actual relevant documents contained in theset of documents identified by the computer as relevant.The training completeness and validation topics will be covered in more detaillater in these guidelines.6

CHAPTER TWOTAR WORKFLOWA.B.INTRODUCTION 8FOUNDATIONAL CONCEPTS & UNDERSTANDINGS . .91.2.KEY TAR TERMS 9TAR SOFTWARE: ALGORITHMS 9a. FEATURE EXTRACTION ALGORITHMS . 9b. SUPERVISED MACHINE LEARNING ALGORITHMS(SUPERVISED LEARNING METHODS) . 10c. VARYING INDUSTRY TERMINOLOGY RELATED TO VARIOUSSUPERVISED MACHINE LEARNING METHODS . . . 10C. THE TAR WORKFLOW . 111. IDENTIFY THE TEAM TO ENGAGE IN THE TAR WORKFLOW . . . 112. SELECT THE SERVICE PROVIDER AND SOFTWARE . . 123. IDENTIFY, ANALYZE, AND PREPARE THE TAR SET . . . 13a. TIMING AND THE TAR WORKFLOW . . 144. THE HUMAN REVIEWER PREPARES FOR ENGAGING IN TAR . 155. HUMAN REVIEWER TRAINS THE COMPUTER TO DETECT RELEVANCY,AND THE COMPUTER CLASSIFIES THE TAR SET DOCUMENTS . . 166. IMPLEMENT REVIEW QUALITY CONTROL MEASURES DURINGTRAINING . . . . 19a. DECISION LOG . . .19b. SAMPLING . . . 20c. REPORTS . . . . . 207. DETERMINE WHEN COMPUTER TRAINING IS COMPLETE ANDVALIDATE . 20a. TRAINING COMPLETION . .21i. TRACKING OF SAMPLE-BASED EFFECTIVENESS21ESTIMATE . .ii. OBSERVING SPARSENESS OF RELEVANT DOCUMENTS RETURNEDBY THE COMPUTER DURING ACTIVE MACHINE LEARNING . . 21iii. COMPARISON OF PREDICTIVE MODELBEHAVIORS . . 22iv. COMPARING TRADITIONAL TAR 1.0 AND TAR 2.0 TRAININGCOMPLETION PROCESSES . . . . 22b. VALIDATION 248. FINAL IDENTIFICATION, REVIEW, AND PRODUCTION OF THE PREDICTEDRELEVANT SET . . . 269. WORKFLOW ISSUE SPOTTING . . . . 27a. EXTREMELY LOW OR HIGH RICHNESS OF THE TAR SET . 27b. SUPPLEMENTAL COLLECTIONS . . . . 28c. CHANGING SCOPE OF RELEVANCY . . . . .28d. UNREASONABLE TRAINING RESULTS . . 297

A. INTRODUCTIONTAR can be used for many tasks throughout the Electronic DiscoveryReference Model (EDRM), from information governance to deposition and trialpreparation, which are discussed in Chapter Three. This chapter focuses on the useof TAR to determine relevancy of documents. To be more specific, the chapter focuseson a suggested workflow by which a human reviewer works with a computer that canbe taught to classify relevant and nonrelevant documents in support of documentproduction obligations. When the human training and computer review are complete,the documents capable of being analyzed will be classified into two piles: the predictedrelevant set, which may have been reviewed by humans or may be unreviewed butpredicted to be relevant (i.e., documents subject to potential production) and thepredicted nonrelevant set, which are typically not reviewed by humans (i.e.,documents not subject to potential production). 10Under this workflow, a human reviewer will have reviewed, or will have theoption to review, the predicted relevant set prior to production. The documents in thepredicted nonrelevant set typically are omitted from human review based on theclassification decisions made by th

Technology assisted review (referred to as "TAR," and also called predictive coding, computer assisted review, or supervised machine learning) is a review process in which humans work with software ("computer") to train it to identify relevant documents. 2.