Validation In M&S - UP

Validation process in M&S: An overview of validation techniques Zafeiris kokkinogenis Intelligent Systems, Interaction and Multimedia Seminar 2012/2013

Outline Introduction What is Validation? Initial considerations Levels of validity Reasons for a model to fail The 7-step approach Decision –making: Who decides if the model is valid? Real & Simulation World Relationships Validation Techniques Data Validity Conceptual model Validation Computerized model verification Operational Validity Graphical Comparisons of Data Confidence intervals Hypothesis tests AI simulation model validation approach General procedure for validating an agent-based simulation Conclusions?

Introduction Use of a simulation model is a substitute for experimentation with the actual system (existing or proposed), which is usually disruptive, not costeffective, or simple impossible. If the model is not a “close” approximation to the actual system, any conclusions derived from the model are likely to be erroneous and may result in costly decisions being made Validation should and can be done for all models , independently the existence or not of the real system

What is Validation? “Substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model” (Schlesinger et al. 1979) That is, provide evidence to support the prove that the model is an accurate approximation of the system.

and Verification? Ensuring that the computer program of the computerized model and its implementation are correct.

Some more definitions Model accreditation determines if a model satisfies specified model accreditation criteria according to a specified process Model credibility is concerned with developing in users the confidence they require in order to use a model and in the information derived from that model

Some initial considerations A “valid” model can be used to make decisions similar to those that would be made if it were feasible and cost-effective to experiment with the system itself A model should be developed for a specific purpose and its validity determined with respect to that purpose The difficulty or not of the validation process depends on the system’s complexity and on whether a version of such system currently exists

and some more A simulation model of a complex system can only be an approximation to the actual system Validation is not something to be attempted after the simulation model has already been developed, and only if there is time and money remaining.

Levels of validity Three levels of validity: – replicative, predictive, structural validity (Zeigler 1976) – the model is able to reproduce given data, unknown data or the original structural relationships Two instances: – Behavioral validity: overall system’s input-output behavior – Structural validity: Internal structure of the model’ validity

Reasons for a model to fail Assumptions over system component, interactions, input data – Lack of knowledge (usually proposed system or not observable) – Missing data/Data distribution Simplifications – modeling complex processes as single processes – Process has insignificant impact on the rest of the model Omissions Limitation – Modeling software – Data

The 7-step approach Law 1 Formulate the Problem 2 Collect Information/Data and Construct Conceptual Model 3 Is the Conceptual Model Valid? · If errors or omissions are discovered in the conceptual model, which is almost always the case, then the conceptual model must be updated before proceeding to programming in Step 4. The substantiation that a model is valid is Step 4. Program the Model considered to be a process and is part of · Program the conceptual model in a commercial the simulation life cycle simulation-software package or in a generalNo Step 5. Is the Programmed Model Valid? Yes 4 Program the Model 5 Is the Programmed Model Valid? No Yes 6 Design, Conduct, and Analyze Experiments 7 Document and Present the Simulation Results purpose programming language (e.g., C, C , and Java). · Verify (debug) the computer program. Figure 1: A Seven-Step Approach for Conducting a Successful Simulation Study · If there is an existing system, then compare simulation model output data for this system with the comparable output data collected from the actual system (see Step 2). This is called results validation. · Regardless of whether there is an existing system, the simulation analysts and SMEs should review the simulation results for reasonableness. If the results are consistent with how they perceive the system should operate, then the simulation model is said to have face validity. · Sensitivity analyses should be performed on the programmed model to see which model factors have the greatest impact on the performance measures and, thus, have to be modeled carefully. Step 6. Design, Conduct, and Analyze Experiments Law (2005) · For each system configuration of interest, decide on tactical issues such as run length, warmup period, and the number of independent model replications.

Who decides if the model is valid? 1. Model development team makes the decision whether the model is valid. 2. Users of the model are the decision-makers abound model’s validity. 3. IV&V; A third party of experts decides if the model is valid 1. Concurrent validation process 2. IV&V is conducted after the development of the model 4. Scoring model

The modelling process A simplified version Sargent (2010)

Validation Techniques (1) Animation: Display model’s operational behavior Comparison to Other Models: Simulated model output vs. known models output Degenerate Tests: Degeneracy of the model’s behavior for certain input and parameter values Event Validity: Event occurrence in simulation vs. real system Extreme Condition Tests: Model’s behavior should be plausible for any extreme and unlikely combination of levels of factors in the system. Face Validity: Experts are asked whether the model or its behavior is reasonable Historical Data Validation

Validation Techniques (2) Historical Methods: Rationalism, Empiricism, Positive economics Empirical Validity: Uses statistical measures and tests to compare model’s key figures with the reference system. Internal Validity: amount of (internal) stochastic variability in the model Multistage Validation Operational Graphics: Plotting various performance measures Parameter Variability - Sensitivity Analysis: Effect of different input/parameters on output/model behavior Predictive Validation: System’s behavior vs. model’s forecast Traces: Follow behavior of entities in the model and determine if its logic is correct Turing Tests

Data Validity Data are necessary for: – Building the conceptual model – Validating the model – Performing experiments with the model Data must be appropriate (format), accurate, unbiased, sufficient and data transformation made correctly – data disaggregation. Collect and maintain data Test collected data for consistency Screening data for outliers

Conceptual model Validation Theories and assumptions are correct – Linearity, I.I.D, Distribution types (Poisson,etc) Model’s representation of the problem entity is reasonable – Model’s structure – Logic, Mathematical, Causal relationships Applicable statistical methods – Curve fitting – Estimation parameters – Plotting data to determine if are stationary Each sub-model and overall model must be evaluated – Face validation: Flowchart or model equation – Traces: Tracking of entities to guarantee the logic is correct.

Computerized model verification Simulation language Vs. General purpose programming language Two basic approaches for testing simulation software: – Static testing Structured Walkthroughs, correctness proofs – Dynamic testing Traces , Input-Output investigation, Internal consistency checks, Reprogramming critical components

Balci (2001)

Operational Validity Simulation model’s output behavior is accurate as required for the model’s intended purpose over the domain of application Major issue is whether the system is observable – Possible to collect data on operational behavior Sargent(2010) Classification of the validation techniques in operational validity

Operational Validity - Explore model behavior Examine the model’s output behavior sensitivity analysis or other techniques. – Qualitative analysis Direction of the output behavior Magnitudes are reasonable – Quantitative analysis Both direction and precise magnitudes of the output Sensitivity analysis – mathematical: nominal range sensitivity analysis, breakeven analysis, difference in log odds ratio, and differentiation – statistical: regression analysis, analysis of variance, Fourier amplitude sensitivity test, mutual information index Graphical comparisons of data Meta-modeling Design of experiments

Operational Validity – Comparison of Output Behavior Compare simulation model output behavior to either the system output behavior or to another known model 1. Graphs for subjective decision-making 2. Confidence intervals for objective decision-making 3. Hypothesis tests for objective decision-making

Graphical Comparisons of Data Histograms, Box plots, Scatter plots(behavior) – Measures: Mean, variance, maximum, distribution, times series – Relationships between two measures of a single or more variables Sargent Figure 6: Reaction Time Figure 7: Disk

Confidence intervals Model’s range of accuracy – Confidence intervals (c.i.) – Simultaneous confidence intervals (s.c.i) – Joint confidence region (j.c.r) To built the range of accuracy a statistical technique and data collection method must be developed Univariate statistical techniques: c.i Multivariate statistical techniques: s.c.i, j.c.r Build range of accuracy with lengths of c.i and s.c.i as small as possible

Hypothesis tests For the acceptable range of accuracy under the set of experimental condition: H0 : Model is valid H1: Model is invalid Type I error: reject H0 when model is valid – P(Type I error) α : Model builder’s risk Type II error: accept an invalid model – P(Type II error) β : Model user’s risk (to be kept small!) Pa (p-value): Probability of acceptance a valid model

AI simulation model validation approach Are traditional model validation approaches longer applied to current AI simulation models? AI simulation models have their own features that distinguish them from conventional models – uncertain input parameters, unavailable referent data, complexity, adaptability, training, learnability, convergence, and generalization AI simulation model validation approach: – Sensitivity analysis – Uncertainty analysis – Output validation

AI simulation model validation approach (2) he Fourth International Conference on Machine Learning and Cybernetics, Guangzhou, 18-21 August 2005 Sensitivity Analysis put is produced. This work begins after we ensitivity analysis. Uncertainty Analysis In what ways do different input values and model variables affect y - r ? Comparing a model with its real system determines how closely its capabilities come to supplying the behavior necessary to achieve the user s objectives. This comparison determines the validity of a model for the user s purposes. According to the availability of the real data, we can use different validation methods to perform output validation [11]. Figure 5 Output presents the Analysis output validation process. es of uncertainty in AI simulation models analysis is to quantify the probability of as a result of the input uncertainty. ysis includes uncertainty characterization Figure 5. Output validation process for AI simulation models Herein a hypothetical example taken from reference