Verification And Validation Of Case-Based Systems
ExpertSystems WithApplications, Vol.6, pp. 57-66, 1993Printed in the USA.0957--4174/93 5.00 .00 1993PergamonPressLtd.Verification and Validation of Case-Based SystemsD A N I E L E. O ' L E A R YSchool of Business, University of Southern California, Los Angeles, CAAbstract--Verification and validation of artificially intelligent systems has been thefocus of substantialrecent research. However, little attention has been given in the literature to verification and validationfor case-based systems. The unique structure of case-based systems is used to raise new validationissues, develop new approaches to generating comparative solutions for validation purposes, andinvestigate new approachesfor examining the quality of the case base. In addition, this article presentsnew statistical and structural approaches designed to exploit unique aspects of case-based reasoningfor verification purposes.1. I N T R O D U C T I O Nis not validated, then it m a y not make the desired quality of decisions.VIRTUALLY ALL THE RESEARCH in verification andvalidation has been focused on rule-based systemsrather than other knowledge representations, such ascase-based systems (Gupta, 1991). Rule-based systemsmake their judgments from rules, while case-based systems make their judgments from previous cases or experience.Thus, the purpose of this article is to summarizeexisting approaches, develop new approaches, and raisenew concerns for the verification and validation of casebased systems. To accomplish those activities, some ofthe unique factors of case-based systems are elicited.Those factors are then used as a basis for developingnew approaches for verification and validation.1.2. Basis of Verification and ValidationVerifying and validating are accomplished by comparing what is expected to what is present. In verification, the basis for these comparisons is the knowledgerepresentation and the knowledge stored in those representations. Typically, the case knowledge is represented using, for example, frames. Thus, these comparisons m a y include, for example, the structure of thecases (e.g., n u m b e r of slots and n u m b e r of slots filledwith meaningful information), the structure of the interaction of the cases (e.g., frames typically use treestructures), and statistically based expectations (e.g.,distribution of various case parameters). These are eachdiscussed in more detail later in the article.Learning mechanisms can impact the case-basedsystem's performance and, thus, validation results. Asa result of those learning processes, typically, case-basedsystems add cases to the case base as the system learns.Thus, an important issue in validating case-based systems is the impact on future system performance ofadding cases from current experience to the case basefor future use. Critical questions include, does comparative performance change based upon the order inwhich cases are added to the case base?In validation, often h u m a n experts are the comparative basis. However, experts can be expensive or unavailable. Thus, one of the focuses of this article is thegeneration of alternative sources for comparison.Mathematical programming and statistics are developed as sources of comparative solutions.Examination of the cases in the case base is a critical1.1. Role of Verification and ValidationVerification tests are aimed at "'building the systemright," and validation tests are aimed at "building theright system." Thus, verification examines issues suchas ensuring that the knowledge in the system is represented correctly, while validation examines procedures to ensure the system makes correct decisions.Verifying and validating play critical roles in thedevelopment and implementation of case-based systems. A priori, if the system is not verified then theremay be errors in the case representations. If the systemPortions of this article weredevelopedwhile visitingBond University,the School of Computing Science, Gold Coast, Queensland 4229,Australia.Requests for reprints should be sent to Daniel E. O'Leary, Schoolof Business, University of Southern California, Los Angeles, CA90089-1421, USA.57
58D. E. O'Learyconcern to the quality of the decisions and, thus, tovalidation. This article uses statistical and similaritymeasures to examine the quality and diversity of thecase base.1.3. This ArticleThis article proceeds as follows. Section 2 providesmore detailed definitions and discussions of verificationand validation. Section 3 reviews the previous literatureon verification and validation of case-based systems.Section 4 argues that case-based systems are differentthan other types of systems and thus should require aunique approach to verification and validation. Thesefactors provide the basis of the remainder of the article.Sections 5 and 6 discuss verification issues. Section 5analyzes some domain-independent statistical approaches for the verification of case-based systems.Section 6 investigates some structural-based approachesthat can be used to verify case-based systems. Sections7-11 present validation approaches developed for casebased systems. Section 7 finds that the sequential addition of cases can cause the development of differentsolutions to the same problems, with different sequential processing of the cases. This is critical to both development and verification and validation of case-basedsystems. Alternative methods of generating solutionscan be useful when expert time is limited, costly, orineffective. As a result, Section 8 presents a mathematical programming approach that can be used togenerate solutions to compare to those of case-basedsystems. Section 9 discusses the use of statistical models,also to generate comparable solutions. Test cases canbe used to assist in the validation of the system. Thus,Section l0 presents an approach using genetic algorithms to generate test cases. The diversity of the casesin a case base can impact the quality of solutions generated. Thus, Section l I develops approaches to testthe similarity of cases so that the diversity of the casesin the case base can be assessed. Finally, Section 12provides a brief summary of the article.2. VERIFICATION AND VALIDATIONThis section presents some definitions of verificationand validation. Those definitions are then briefly analyzed in terms of what they mean for case-based systems.2.1. VerificationVerification was defined by Adrion, Branstad, andCherniavsky (1982) as "the demonstration of the consistency, completeness and correctness of the software."This definition often is supplemented to include redundancy to provide greater specificity to the notionof completeness.Verification's dependence upon software indicatesthat the specific nature of case-based systems needs tobe elicited to perform verification. Because verificationis software based, that is, for case-based systems, verification is concerned with exploiting the software representations of, for example, cases and relationshipsbetween cases to establish tests for consistency, completeness, correctness, and redundancy.Consistency, in case-based systems refers to parallelimplementation of parallel structures, whether thosestructures are words or relations between cases, suchas trees. Completeness is concerned with the possibilitythat knowledge or cases are omitted. Correctness refersto determination of whether or not there are any ascertainable errors in the knowledge, for example, circularity in a structure that is supposedly acyclic. Redundancy addresses the issue of duplication of knowledge, for example, duplicate versions of the same case.Verification also can be dependent upon the domain. By exploiting knowledge of the domain, we canverify the knowledge in a case-based system. Considerthe example of a system that has a number of casesthat are different living rooms. Metaknowledge couldbe used to examine the cases to determine that therewas something incorrect with a case that had a bathtubin the living room. There probably would be redundantknowledge in the case if the living room case containedtwo couches. Similarly, the case of a living room without a couch probably could be incomplete. Finally, itwould be inconsistent if in the case two physicallyidentical items were labeled with the same name, forexample, couch. Such domain-dependent approachesare outside the scope of this article because they requirespecification of metaknowledge from the specific domain.2.2. ValidationAdrion et al. (1982) indicate that validation is the determination of the correctness of the final program orsoftware produced from a development project withrespect to the user needs and requirements. In manysoftware development projects, the needs and requirements can be established a priori. However, case-basedsystems are used in situations where the problem is notwell structured enough to develop a rule-based system.Thus, for case-based systems there are likely to be fewsituations where specifications can be elicited a priori.As a result, validation may have to employ differentbases of comparison rather than requirements. Theprocess of validation for case-based systems may besimilar to the generation of the needs and requirementsfor other artificial intelligence (AI) systems. For example, validation may take the form of comparing anexpert to the system for different test cases. Alternatively, there may be other approaches that could exploitthe unique characteristics of case-based systems to
Verification and Validation of Case.Based Systemsgenerate comparison bases. These approaches couldbe cost beneficial by limiting the use of experts in validation processes. The development of alternativemodels of comparison is a primary focus of this article.3. VERIFICATION AND VALIDATION OFCASE-BASED SYSTEM: PREVIOUS STUDIESOne purpose of this article is descriptive. This sectionsummarizes some of the previous descriptions of verification and validation efforts of case-based systems.Unfortunately, there have been a limited number ofstudies summarized in the literature (the literature oncase-based systems is also limited, in general). In addition, in many situations the discussion of those effortsis limited to a few sentences or paragraphs. Further,typically the focus of those discussions has been onvalidation. There are virtually no descriptions of verification efforts for case-based systems.However, it is not surprising that few have discussedverification and validation efforts. The purpose of muchof the research on case-based systems, to date, has beento explore various developmental aspects of case-basedsystems, such as retrieval or indexing. Thus, the lackof research on verification and validation is not a criticism. Instead, this indicates the importance of developing approaches to assist in both the verification andvalidation of case-based systems.3.1. ProtosPossibly the most extensive validation of a case-basedsystem in the literature is for the system Protos. Protos(Bareiss, 1989; Bareiss, Porter, & Weir, 1988) is a casebased learning apprentice that learns to perform heuristic classification under the guidance of a human expert. The evaluation of Protos was in the area of clinicalaudiology, where it learned to classify hearing disordersfrom featural descriptions in terms of patient symptoms, history, and test results.Because Protos is a classification system, the validation considered the accuracy of its classifications.Accuracy comparisons included comparison to othermachine learning approaches and to human expertsand students.When compared to other forms of machine learning,Protos was apparently very successful, given the set oftest problems. The accuracy of Protos was found to befar superior to the well-known ID3 (Quinlan, 1986)and Cobweb (Fisher, 1987). On a set of 26 test cases,Protos was correct 100% of the time, ID# 29% of thetime, and Cobweb 58% of the time.When compared to human experts and students,the system also did well. While Protos was 100% correcton the test problems, two clinicians had 92 and 81%correct, while students had a mean of 73%.Apparently, Protos faced a reasonably large base of59cases. In the training of Protos, a human expert characterized 7 of the final 50 training cases as unusual.3.2. HYPOOne of the best-known case-based systems is HYPO.HYPO was a system developed by Ashley and Rissland(1988) to analyze trade secret law. The reported validation of HYPO's (Ashley & Rissland, 1988) performance was the comparative analysis of a single "realworld" case by the system. They compared HYPO withwhat the court did on that case and found that thesystem agreed with the court.3.3. Clavier (Mark, 1989)The description of evaluation efforts for Clavier is actually a plan for validation. However, that plan had atleast two components. First, the adaptation oftbe casesby the system would be compared to experts. Thus,one of the components of the system would be tested,probably using a classic Turing test. Second, once thesystem was "working" the overall behavior of the system would be compared to experts for quality of recommendations.3.4. The Battle Planner (Goodman, 1989)To validate The Battle Planner, a randomly selectedset of 10% of the cases were put aside before indexing.Then, those cases were treated as hypothetical battlesituations. Predictions were made regarding the outcome of those situations. Using this approach, the system was found to be 81.3% accurate in the predictionof the victor of the case for land warfare.The validation efforts pointed toward an interestingfinding. Goodman found that the more cases that wereretrieved for analysis of a hypothetical case the betterthe performance of the system. Such a finding mayoccur for a number of reasons. First, it may be thatthe more cases offering feasible solutions the morelikely that the solution space is spanned. Second, itmay be that the more feasible solutions the more likelythat a directly similar case could be found. In eithercase, this provides additional evidence about the importance of the quality of the cases in the case base onthe performance of the system.3.5. SummaryAn analysis of the published reports of the validationof case-based systems indicates that the approachesused to date are similar to those used for other intelligent or expert systems. Typically, the validation ofeach system has used the comparison of the system tohuman experts or machine learning. This comparisonranged from a single case to multiple test cases. In gen-
60D. E. O'Learyeral, the system has been compared to, for example,human experts.4. S O M E U N I Q U E FACTORS O FCASE-BASED S Y S T E M SThe focus of this section is to elicit some of the uniqueaspects of case-based systems and use them as a basisfor eliciting different approaches and concerns to verification and validation. The existence of unique factorsboth indicates a need to establish new approaches andprovides new opportunities.As noted by Ashley and Rissland ( 1988, p. 70), casebased reasoning is used " . . . to capture expertise indomains where rules are ill-defined, incomplete or inconsistent." It can be difficult to investigate the performance of a system in the presence of such difficulties.This suggests that it may be difficult to establish expectations or standards of behavior in case-based systems. In particular, the well-established process, in traditional software engineering, of using a priori specifications may not be feasible.Because cases typically are represented using framesrather than rules, this indicates that verification approaches for case-based structures could exploit theframe structure. In particular, for example, verificationprocesses could exploit the nature of cases within thecontext of the frames. This is important because virtually all of the work on verification is for rule-basedsystems.In addition, case-based systems are unique compared to other types of AI systems in that they oftenadd solved cases to their case base. Previous solutionsbecome part of their experience. Those cases are thenused in the development of future solutions. This is acritical difference from, for example, rule-based systems, where the knowledge base in general is static.Although case-based systems often are designed tocreate new solutions, typically the quality of the solutions is dependent, in large measure, upon the casebase. The quality of the recommendations of a casebased system is dependent upon the quality and quantity of the cases in the case base.In general, a case-based system will be able to generate a better recommendation if it has a larger ratherthan smaller case base (e.g., Ruby & Kibler, 1988;Goodman, 1989; Gaines, 1991 ). It is likely that witha larger base of experience the situations faced by thesystem will have been seen before and thus a case willbe available to assist in solution of the problem. Evenif no identical situation has been seen before, the systemis likely to have solutions that are so-called "near miss"situations.If the cases are highly correlated, then the systemwill be limited in the diversity of solutions it can generate. Thus, it is not unusual that Bradtke and Lehnert( 1988, p. 91 ) find " . . . that the most dramatic factorinfluencing the effectiveness of a case base is the number of unique problem states underlying the case baseencoding."Some of these issues can be addressed using approaches such as the direct analysis of the cases byexperts, comparison of expert solutions to the system,and investigation of the system reasoning (choice ofcases) by experts. However, the focus of this article ison the development of alternative approaches to account for these unique factors.5. D O M A I N - I N D E P E N D E N T VERIFICATIONO F CASES: S T R U C T U R A L A P P R O A C H E SIn this section, no assumptions are made of the domain.Instead, the individual structure of the cases and thestructural relationship between the cases are used toprovide some bases with which to verify the cases. Unlike the next section, which uses statistics to identifyanomalous cases that appeared to be in error, this section uses that structure to identify cases that are inerror. Particular emphasis is placed upon using thosestructures to design verifiable systems or systems thatmitigate the opportunity for the existence of the typesof errors isolated.5.1. ConsistencyErrors in consistency can occur for many reasons. Forexample, people can make errors by misspelling or using different names for the same object, actor, or activity.To design a system to mitigate those types of errors,at the initiation of a new case-based system the usercould be required to list each of the cases and corresponding case attributes before any data is entered.Then, those lists of feasible entries for each of the casesand attributes would be used to fill attribute slots. Developers of different cases could choose from the listsof attributes for the content for a specific attribute. Asseen below, these lists also can be used for other purposes.5.2. RedundancyRedundancy errors occur if the user is able to developthe same case more than once. The situation of redundant cases can cause difficulties, primarily inmaintenance situations. One version of the ease couldbe revised while another is r.ot. This could cause confusion and possibly errors if the unrevised case wasused by the system. The possibilities of redundant casescan be mitigated if the user is required to establish alist of the specific cases prior to actually establishingthe data within each case. As new cases are added, they
Verification and Validation of Case-Based Systemswould be added to the list of feasible cases. Each casewould then be referenced by its specific name. Thus,a single version of any given case would be permitted.Redundancy errors also occur if the user is able toenter the same attribute information into two or moreattribute spaces for a given case. There are two basicsituations: two attribute instances in the same attributeslot and the same two attribute instances in differentslots in the same case. In the first situation, maintenance of one entry but not the other m a y cause ambiguity as to the proper contents. In the second situation, the attribute instance m a y be an inappropriateoccurrence for that attribute instance.The first situation could be eliminated if each attribute slot has space for only one entry. The second situation could be eliminated if the attribute instancescould only come from eligible lists, as discussed earlier.5.3. Completeness61TABLE 1Sample CuesName: M-MEAL774 Isa: ((M-MEAL )Category: INDIVIDUALSlotsACTUAL-RESULTS: NILCHARACTERS: (?HOST ?GUESTS?PARTICIPANTS)CONSTRAINTS: ((C-LIMITED-SPACE778 )DEFINED SLOTS: NILDESCRIPTOR: NILEXPECTED-RESULT: NILFOLLOW-UP: NILGOALS: ((E-EAT776 (S-HUNGER777 )GUESTS: (,JLK,S-GROUP HOST: (,JLK, ORDER: NILPARTICIPANTS: (?HOST ?GUESTS)SETTING: (,JLK,S-HOUSE STEPS: NILTIME: NILName: M-MEAL80bIsa: ((M-MEAL )Category: INDIVIDUALThere are at least two types of completeness errors inverification: missing cases and missing attributes forcases. There are some steps that can be taken to mitigatethese errors.Completeness of the cases m a y be difficult to assess.However, having the users establish a list of cases theyintend to enter into the system prior to entering dataabout the cases could facilitate knowing about completeness. Thus, if a case was on the list but not in thecase base this could indicate an incompleteness error.Completeness of the attributes may also be a difficultissue. There are at least two approaches to analysis ofcompleteness. First, each case could be required to haveslots for the same n u m b e r of attributes. As an exampleo f this, the two cases in Table 1 have different numberso f attributes. " T I M E " is in the first but not the second.Second, users could be required to indicate then u m b e r of attributes, of the total set, that would haveinformation in them for a specific case. For example,in Table 1 for the first case seven attributes would bespecified. I r a specific attribute slot had no informationfor a given case, then a variable representing that alternative would be entered into the slot (e.g., "nil" asin Table l ).Completeness tests could also make use of the listsdeveloped for consistency. In addition, after the userput the attributes onto a list it would be possible toreconcile the lists and the completed cases to determineif each o f the attributes put onto lists were used. Ifthere was an attribute that was listed but not used, thatcould indicate that one of the cases omitted appropriateinformation.Completeness in the verification of cases requiresthat the cases that are planned to be included in thesystem are included and that for each of the cases acomplete case specification is given.SlotsACTUAL-RESULTS: (ACTUAL-RESULT80 CHARACTERS: (?HOST?GUESTS?PARTICIPANTS)CONSTRAINTS: ((C-COST11 )DEFINED SLOTS: NILDESCRIPTOR: (MEAL-DESCRIPTOR80 EXPECTED-RESULT: (EXPECTED-RESULT80 FOLLOW-UP: NILGOALS: ((S-HUNGAR80 (E-EAT80 )GUESTS: (*JLK* S-PARENTS HOST: (,JLK, ORDER: ((SC-SALAD80 (SC-MAIN-COURSE80 )PARTICIPANTS: (?HOST ?GUESTS)SETTING: (,JLK,S-HOUSE STEPS: ((SC-SALAD80 (SC-MAIN-COURSE80 )"From Proceedings:Case-BasedReasoningWorkshop(p. 28)by J. Kolodner, 1988, San Mateo, CA: Morgan Kaufmann. Copyright 1988 Morgan Kaufrnann. Reprinted with permission.b From Proceedings:Case-BasedReasoningWorkshop(pp. 2930) by J. Kolodner, 1988, San Mateo, CA: Morgan Kaufmann.Copyright 1988 Morgan Kaufmann. Reprinted with permission.5.4. CorrectnessThe role of graph theoretic structure permeates casebased reasoning systems. Ashley and Rissland (1988a)demonstrate two different "claim lattices" (in solutiongeneration). The structure of those lattices is that of atree. Navinchandra (1988) illustrates that the objecthierarchy used for matching is a tree. The causal structures generated by the system discussed in Navinchandra (1988) are acyclic graphs. If it is known a priorithat the solution or case relationships will be tree oran acyclic graph, then that structure can be exploitedto ensure that the structure is correct.There are a n u m b e r of characteristics of trees thatallow for rapidly determining if a structure is a tree.For example, each child has at most one parent. Another check is to ensure that a node in the solution
62does not have an arc going from itself to itself, whichwould imply inheritance from itself. Alternatively, thestructure may be that of an acyclic network. In thatsituation, the resulting structure must be free of cyclesfor the system to be correct.D. E. O'Learya nonnil value for host. If 39 were J L K ) and 1 was JKL), then the case with the single occurrence wouldwarrant further investigation. This could assist in thelocation of incorrectness problems.6.3. Analysis6. D O M A I N - I N D E P E N D E N T VERIFICATIONO F CASES: STATISTICAL A P P R O A C H E SStatistical approaches to domain-independent verification exploit statistical methods as the basis of establishing expectations. The general approach is to determine if there is any anomalous behavior exhibitedin statistical summaries of characteristics of cases.Anomalies indicate that something may be wrong and,thus, should be further analyzed. The existence ofanomalies is established by developing and using statistical distributions.There are two different statistical approaches tofinding anomalies. Their implementation in a specificsystem would require accounting for the specific set ofcases. As a result, the analysis presented here is designedfor the example cases presented here but can be generalized. To illustrate these approaches, consider theexample case in Table I.6.1. Distribution of Slot Contents Per CaseAnalysis of slot contents can provide insight into theexistence of anomalous behavior. One approach is todevelop a distribution of the number of slots in eachcase with, for example, nil contents. Then, the distribution could be analyzed to determine if any of thecases were exhibiting unusual behavior.The first example case has 15 generic slots for cases,of which 8 have the value nil and 7 have values differentthan nil. At the extreme, if all cases but this case had,say, either 0, 1, or 2 values of nil then that would suggestthat this specific case was unusual. Such an approachcould help locate incomplete and underspecified cases.6.2. Distribution of Contents Per Slot Across CasesIn addition, distributions of slot content could be developed, for example, a distribution of nil values foreach slot could be developed. If there was a slot wherethe number of values nil was very large, then that couldsuggest anomalous behavior because it would indicatethat slot is only rarely considered important. Becausethis approach would identify slots where nil was thevalue, it would suggest that those slots may be incomplete across many different cases. For example, both"FOLLOW-UP" and "DEFINED SLOTS" contain thevalue nil for both cases.Another approach would be to determine distributions for slot contents for each individual slot. Inthe example, assume that there were 40 cases that hadIn each situation, these distributions could be analyzedas in the extreme cases listed here and using establishedforms of analysis. For example, outlier analysis couldbe used to find those points in those distributions thatappear to come from different distributions and thusmay deserve further investigation. In addition, statistical tests of significance (e.g., t test, etc.) could be usedto determine anomalies.7. S E Q U E N C E EFFECTS OF ADDING CASEST O T H E CASE BASEOne of the critical conditions of validation is the abilityto duplicate the response of a system under similarconditions. Unfortunately, the behavior of case-basedsystems can vary depending upon the order in whichcases are processed.Case-based systems use a number of approaches forchoosing the best match from the case base for thesolution of the case under consideration. After the system proposes a solution, it then adds the new case toits case base for further use. Validating these systemsis difficult because, as shown in this section, the orderin which validation test cases are added to the systemcan influence the system's proposed solutions.7.1. Solutions Are a Function of Order of CasesEncounteredConsider a case-based system that adds cases to its casebase as it solves them. Assume that one of the primarybases of solution is the number of attributes that thesituation under consideration has in c o m m o n withsome other case in the case base. Suppose that twooutcomes, x and y (captured in the first slot), are possible. Next, suppose that there are six additional slotsthat capture attributes of the individual cases. It is assumed that attribute l is in the first slot, and so forth.Suppose that the initial case base consists of the twocases, 1. (x; a, b, c, d , f , m) and 2. (y; a, b, c, h, g,m). If the next case seen is 3. (-; a, b, c, d, e, n), thenit is resolved as 3. (x; a, b, c, d, e, n). As result, if thenext case is 4. (-; a, b, c, h, e, n) then it will be resolvedas4. (x; a, b, c, h, e, n).However, if case 4 is seen before case 3 then thefollowing scenario will occur. Case 4. (-; a, b, c, h, e,n) will be resolved as 4. (y; a, b, c, h, e, n), while case3. (-; a, b, c, d, e, n) becomes 3. (y; a, b, c, d, e, n).Clearly, the sequence can have a major impact onthe system. Solutions developed by the system can be
Verification and Validation of Case-Based Systemsan artifact of the order in which they are processed.This presents a major concern in validation. Althoughthe use of test cases may find that the system agreeswith experts or other models, this may be simply afortuitous function of order.7.2. Impact of Order EffectsCase-based systems add cases as they encounter them.Because the solutions they provide are, in part, a function of their case base, the addition o f cases to the casebase can change the solutions they provide. Thus, th
new approaches for verification and validation. 1.1. Role of Verification and Validation Verification tests are aimed at "'building the system right," and validation tests are aimed at "building the right system." Thus, verification examines issues such as ensuring that the knowledge in the system is rep-
series b, 580c. case farm tractor manuals - tractor repair, service and case 530 ck backhoe & loader only case 530 ck, case 530 forklift attachment only, const king case 531 ag case 535 ag case 540 case 540 ag case 540, 540c ag case 540c ag case 541 case 541 ag case 541c ag case 545 ag case 570 case 570 ag case 570 agas, case
verification and validation. 1.2 PURPOSE This System Validation and Verification Plan provide a basis for review and evaluation of the effectiveness of the AIV program and its proposed elements. In addition it is an input to the lower level verification. In this document it is proposed a scenario for the full requirements traceability throughout a
Validation of standardized methods (ISO 17468) described the rules for validation or re-validation of standardized (ISO or CEN) methods. Based on principles described in ISO 16140-2. -Single lab validation . describes the validation against a reference method or without a reference method using a classical approach or a factorial design approach.
Cleaning validation Process validation Analytical method validation Computer system validation Similarly, the activity of qualifying systems and . Keywords: Process validation, validation protocol, pharmaceutical process control. Nitish Maini*, Saroj Jain, Satish ABSTRACTABSTRACT Sardana Hindu College of Pharmacy, J. Adv. Pharm. Edu. & Res.
- Validation (§ 117.160) - Verification that monitoring is being conducted - Verification that corrective action decisions are appropriate - Verification of implementation and effectiveness (§ 117.165) Calibration, product testing, environmental monitoring, review of records - Reanalysis
Independent Verification and Validation (IV&V) Verification by independent authorities necessary for but not limited to requirements that are safety-critical or of high-security nature Independent verification and validation is defined by three parameters: Technical, Managerial und Financial Independence
System verification refers to checking that a delivered system meets its requirements, usually by testing (although other verification methods are employed where testing is costly or impossible). Note that the validation activity (as defined here) must precede verification, as pointed out in O'Grady.2 Unfortunately,
Judul : Buku Panduan Profesi Ners Stase Keperawatan Keluarga 2. Penyusun : Puji Lestari, S.Kep., Ns., M.Kes (Epid) Ns. Puji Purwaningsih, S.Kep., M.Kep. 3. Unit Kerja : Prodi Pendidikan Profesi Ners Ungaran, 3 September 2020 Ketua Program Studi Ns.Sukarno, S.Kep., M.Kep NIDN. 0624128204 . viii . ix KATA PENGANTAR Puji dan Syukur kehadirat Tuhan Yang Maha Esa yang telah mencurahkan rahmat dan .
