TecHnology AssessmenT Design Handbook
HANDBOOKTechnologyAssessmentDesignHandbookHandbook for Key Steps and Considerationsin the Design of Technology AssessmentsGAO-20-246GDecember 2019
ContentsPrefaceChapter 11The Importance of Technology Assessment Design61.1 Reasons to Conduct and Uses of a Technology Assessment1.2 Importance of Spending Time on Design68Technology Assessment Scope and Design82.1 Sound Technology Assessment Design2.2 Phases and Considerations for Technology AssessmentDesign2.2.1 GAO Technology Assessment Design Examples9914Approaches to Selected Technology Assessment Design andImplementation Challenges183.1 Ensuring Technology Assessment Products are Useful forCongress and Others3.2 Determining Policy Goals and Measuring Impact3.3 Researching and Communicating Complicated Issues3.4 Engaging All Relevant Stakeholders19202021Appendix IObjectives, Scope, and Methodology22Appendix IISummary of Steps for GAO’s General Engagement Process35Appendix IIIExample Methods for Technology Assessment38Appendix IVGAO Contact and Staff Acknowledgments44Chapter 2Chapter 3Page iGAO-20-246G Technology Assessment Handbook
TablesTable 1: Summary of GAO’s Technology Assessment ProcessTable 2: Examples for Technology Assessment Objectives thatDescribe Status and Challenges to Development of aTechnologyTable 3: Examples for Technology Assessment Objectives thatAssess Opportunities and Challenges that May Resultfrom the Use of a TechnologyTable 4: Examples for Technology Assessment Objectives thatAssess Cost-Effectiveness, Policy Considerations, orOptions Related to the Use of a TechnologyTable 5: Challenges to Ensuring Technology AssessmentProducts are Useful for Congress and OthersTable 6: Challenges to Determining Policy Goals and MeasuringImpactTable 7: Challenges to Researching and CommunicatingComplicated IssuesTable 8: Challenges to Engaging All Relevant StakeholdersTable 9: Select Examples of Methodologies for TestimonialEvidenceTable 10: Select Examples of Methodologies for DocumentaryEvidenceTable 11: Select Examples of Methodologies for PhysicalEvidence315161719202021394041FigureFigure 1: Summary of Key Phases and Considerations ofTechnology Assessment DesignPage ii10GAO-20-246G Technology Assessment Handbook
AbbreviationsAICRSEMSOTAS&TSTAATATRLartificial intelligenceCongressional Research ServiceEngagement Management SystemOffice of Technology Assessmentscience and technologyScience, Technology Assessment, and Analyticstechnology assessmenttechnology readiness levelsThis is a work of the U.S. government and is not subject to copyright protection in theUnited States. The published product may be reproduced and distributed in its entiretywithout further permission from GAO. However, because this work may containcopyrighted images or other material, permission from the copyright holder may benecessary if you wish to reproduce this material separately.Page iiiGAO-20-246G Technology Assessment Handbook
Letter441 G St. N.W.Washington, DC 20548PrefaceGAO provides Congress, federal agencies, and the public with objective,reliable information to help the government save money and work moreefficiently. Science and technology (S&T) issues figure prominently inproblems that Congress confronts, and one component of the assistanceGAO provides to Congress is the production of technology assessments(TAs). This TA Design Handbook provides GAO staff and others withtools to consider for supporting robust and rigorous assessments. Thishandbook is particularly important given the need for GAO to provideinsight and foresight on the effects of technologies and correspondingpolicy implications related to a wide range of S&T issues. While otherorganizations—including, previously, the Office of TechnologyAssessment (OTA) and a number of TA organizations elsewhere, such asin Europe—conduct TAs, each has different relationships with itsstakeholders and government bodies. While their TA approaches andconsiderations may vary, some may still find portions of this handbookuseful. We are seeking comments on this draft of the handbook.This handbook elaborates on GAO’s approach to TA design and outlinesthe importance of TA design (Chapter 1), describes the process ofdeveloping TA design (Chapter 2), and provides approaches to select TAdesign and implementation challenges (Chapter 3). The handbookgenerally follows the format of the 2012 GAO methodology transferpaper, Designing Evaluations. 1 Given that GAO is likely to learn from itscurrent expansion of TA work, GAO will review and update this drafthandbook as needed, based on experience gained through ongoing TAactivities and external feedback.GAO has defined TA as the thorough and balanced analysis of significantprimary, secondary, indirect, and delayed interactions of a technologicalinnovation with society, the environment, and the economy and thepresent and foreseen consequences and impacts of those interactions. 2The effects of those interactions can have implications. Recognizing this,GAO has in some of its products included policy options, whichDesigning Evaluations describes designs of program evaluations. See GAO, DesigningEvaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: Jan. 2012).12There is no single agreed-upon typology/taxonomy of or approach to TAs. Examples ofdifferent TA approaches found in the literature include: strategic, early-warning, futureoriented, classical or expert, and participatory. Expert TAs may emphasize expertknowledge, and participatory TAs may emphasize stakeholder and public involvement.Page 1GAO-20-246G Technology Assessment Handbook
policymakers could consider in the context of a given technology andpolicy goal. In this context, policy goals serve to guide the development ofpolicy options by stating the overall aim of the policy options, and helpingto identify the landscape and scope of policy options. Policy options canbe defined as a set of alternatives or menu of options (including the statusquo) that policymakers, such as legislative bodies, government agencies,and other groups, could consider taking. GAO is exploring approaches tomaking policy options a more standard feature or component of TAs. Inthis handbook, we include considerations related to the development ofpolicy options that TA teams may wish to consider at each phase of TAdesign.Technology Assessments at GAOGAO has a history of doing S&T related work,including audits of federal S&T programs. Infiscal year 2018, GAO provided 34congressional committees with nearly 200products, including technology assessments,covering a wide range of science, technology,and information technology issues, includingcybersecurity. In 2018, Congress encouragedthe formation of a Science, TechnologyAssessment, and Analytics (STAA) teamwithin GAO, recognizing that the scope oftechnological complexities continues to growsignificantly and there is need to bolstercapacity of, and enhance access to, quality,independent science and technologicalexpertise for Congress. STAA was formallycreated on January 29, 2019.Source: STAA Initial Plan. GAO-20-246GIn the United States, the Technology Assessment Act of 1972 establishedOTA, which was an analytical support agency of the Congress, but wasdefunded in 1995. In 2002, Congress asked GAO to begin conductingTAs, and in 2008, a permanent TA function was established at GAO. In2019, the Science, Technology Assessment, and Analytics (STAA) teamwas created at GAO. STAA has taken a number of steps to account forthe unique requirements of TAs and related S&T work to meet the needsof Congress.GAO TAs share some common design principles with GAO’s generalaudit engagement process, which is centered around intentional andpurpose-driven design. 3 While general design principles are sharedacross GAO’s product lines, TAs are distinct from other GAO productlines, such as performance audits, financial audits, and other routine nonaudit products. The specialized content of TAs, their scope, and theirpurpose, warrant some different considerations. Table 1 highlights somesimilarities and differences between TAs and other GAO product lines,including where TAs follow aspects of GAO’s general audit engagementprocess, and where TAs may further emphasize certain steps or require3For example, GAO’s general audit engagement process includes a robust initiation anddesign processes that consider factors such as: stakeholder interests, the current state ofknowledge, and relevant and appropriate methodological considerations in defining andinvestigating appropriate research questions. Also part of GAO’s general auditengagement process is internal message development and agreement, along withexternal review processes. Design decisions are implemented and revisited throughoutthe audit engagement process.Page 2GAO-20-246G Technology Assessment Handbook
additional steps during the engagement process. 4 Not all steps have beenincluded in Table 1.Table 1: Summary of GAO’s Technology Assessment ProcessSteps in plain text are process steps for both general audit and TA products. Steps in bold italics are either additional process steps ora particular emphasis for TAs.PhaseStepsInitiation Design Message development External review Discussion with congressional requesters regarding scope and focus of the engagementConsideration of technology state, breadth of stakeholder expertise, and potential policyimplicationsConsideration of whether policy options are appropriate for inclusionPerformance of initial researchConsideration of relevant sections of GAO’s Quality Assurance Framework and GAO methodologicaland technical standards and guidesConsultation with GAO subject matter experts and internal stakeholders, as neededDiscussion with agency officialsIdentification of and consultation with external experts, such as science, policy, and industryaexperts, who may also serve as external reviewersIdentification of initial policy options, if appropriateCollection and analysis of evidenceAssessment of evidence and research resultsDevelopment of draft findingsOngoing engagement with external expertsbPerformance and discussion of the results of policy options assessment, if appropriateRequest views from relevant third parties, if applicable, and request comments from relevant federalagencies, as appropriateRequest comments from external expertsSource: GAO analysis of GAO’s product line processes. GAO-20-246GaGAO has a standing task order contract with the National Academies of Sciences, Engineering, andMedicine. GAO can interact with National Academies personnel to help GAO identify experts onvarious scientific topics and also can leverage National Academies assistance to convene GAOexpert meetings.bInstead of recommendations, TAs may consider the inclusion of policy options.We expect to continue to regularly seek input and advice from externalexperts related to the TA Design Handbook initiative, as well asthroughout the conduct of GAO TAs. While the primary audience of thishandbook is GAO staff, we expect that other organizations engaged orinterested in TAs will find portions of this handbook useful. For example,4Products from other product lines may emphasize these elements as well, depending onengagement needs.Page 3GAO-20-246G Technology Assessment Handbook
these organizations could use the handbook to gain insight into GAO’s TAdesign approaches, as well as use aspects of GAO’s TA designapproaches that they deem helpful. We will accept comments on thishandbook at TAHandbook@gao.gov for approximately 1 year afterpublication. The handbook seeks to affirm and document GAO’sapproach, and we expect to modify and refine this handbook, as needed,based both on comments received and further experience in conductingTAs that include policy options. We anticipate that the final handbook willcontain additional information and details related to TA design, such aselaborating on specific methodologies that could be applied within thisgeneral design framework, including those designed to identify policyoptions.Below is a summary of the approach we used to identify and documentTA design steps and considerations for this handbook. For moreinformation, please refer to Appendix I: Objectives, Scope, andMethodology. Reviewed select GAO documents, including Designing Evaluations(GAO-12-208G), published GAO TAs, select GAO products utilizingpolicy analysis approaches to present policy options, and other GAOreports Reviewed select Office of Technology Assessment reports Reviewed select Congressional Research Service reports Reviewed select literature regarding TAs and related to developmentand analysis of policy options Held an expert forum to gather experts’ input regarding TA design Considered experiences of GAO teams that have successfullyassessed and incorporated policy options into GAO products, as wellas GAO teams that are incorporating policy options into their TAdesign Collected input from GAO staff who provided key contributions toGAO TAs, regarding challenges to TA design and implementation andpossible solutionsWe conducted our work to develop this handbook from April 2019 toDecember 2019 in accordance with all sections of GAO’s QualityAssurance Framework that are relevant to our objectives. The Frameworkrequires that we plan and perform the engagement to obtain sufficientand appropriate evidence to meet our stated objectives and to discussPage 4GAO-20-246G Technology Assessment Handbook
any limitations in our work. We believe that the information and dataobtained, and the analysis conducted, provide a reasonable basis for anyfindings and conclusions in this product.Timothy M. Persons, PhDManaging DirectorScience, Technology Assessment, and AnalyticsChief Scientist, GAOKaren L. Howard, PhDDirectorScience, Technology Assessment, and AnalyticsPage 5GAO-20-246G Technology Assessment Handbook
Chapter 1: The Importance of TechnologyAssessment DesignChapter 1: The Importance of TechnologyAssessment DesignThis chapter underscores the importance of technology assessment (TA)design, outlining reasons for performing TAs and for spending time on thedesign of TAs. The information presented in this chapter is based onreview of results of a literature search, an expert forum, select GAOreports, and experiences of GAO teams and technical specialists. Formore information, please refer to Appendix I: Objectives, Scope, andMethodology.1.1 Reasons toConduct and Uses ofa TechnologyAssessmentTAs are significant given their increasing importance to policymakers, andthe growing effects of S&T on society, economy, and other areas. Whiletechnological changes can be positive, they can also be disruptive.Therefore, it is critical for Congress to be able to understand and evaluatethese changes, to ensure, for example, national security and globalcompetitiveness. Examples of potential uses of TAs related to enhancingknowledge and awareness to assist decision-making include: Highlight potential short, medium, and long-term impacts of atechnology Elaborate on and communicate the risks and benefits associated witha technology, including early insights into the potential impacts oftechnology 1 Highlight the status, viability, and relative maturity of a technology Plan and evaluate federal investments in S&TGAO TAs are most commonly requested by congressional committees,which may use them to, among other things, make decisions regardingallocating or reallocating resources to address research gaps, support1This may include analyzing and providing information on the costs and benefits of aspecific technology or set of technologies, and their present and potential futurechallenges.Page 6GAO-20-246G Technology Assessment Handbook
Chapter 1: The Importance of TechnologyAssessment Designupdated rulemaking for a regulatory agency, or inform a legislativeagenda or the development of a national strategy. 2Technologies present opportunities and challenges that may vary,depending in part on the policy context in which they are evaluated.Therefore, part of a TA is considering the policy context surrounding agiven technology. GAO may, where appropriate, identify and analyzepolicy options as part of its TAs, which may also include: clarifying andsummarizing policy-related issues and challenges, and providinginformation that can be used for decision-making. In this situation, policyoptions can be defined as a set of alternatives or menu of options(including the status quo) that policymakers, such as legislative bodies,government agencies, and other groups, could consider taking. Policyoptions can be used to articulate a range of possible actions apolicymaker could consider in the context of a given technology andpolicy goal. Policy options do not state what policymakers should do in agiven circumstance with a certain technology. Policy options do notendorse or recommend a particular course of action; they are notrecommendations or matters for congressional consideration, which GAOmakes in its audits. In addition, policy options are addressed to2Examples of research questions and objectives from published GAO TAs include: (1)what is known about the potential effects of geomagnetic disturbances on the U.S. electricgrid; what technologies are available or in development that could help prevent or mitigatethe effects of geomagnetic disturbances on the U.S. electric grid, and how effective arethey; and what factors could affect the development and implementation of thesetechnologies? GAO, Critical Infrastructure Protection Protecting the Electric Grid fromGeomagnetic Disturbances, GAO-19-98 (Washington, D.C.: Dec, 19, 2018); (2) how hasartificial intelligence (AI) evolved over time, and what are important trends anddevelopments in the relatively near-term future; according to experts, what are theopportunities and future promise, as well as the principal challenges and risks, of AI; andaccording to experts, what are the policy implications and research priorities resulting fromadvances in AI? GAO, Artificial Intelligence: Emerging Opportunities, Challenges andImplications, GAO-18-142SP (Washington, D.C.: Mar. 28, 2018); and (3) Identify biometrictechnologies currently deployed, currently available but not yet deployed, or indevelopment that could be deployed in the foreseeable future for use in securing thenation’s borders; determine how effective these technologies are for helping providesecurity to our borders currently or are likely to be in the future; determine the economicand effectiveness trade-offs of implementing these technologies; and identify theimplications of biometric technologies for personal security and the preservation ofindividual liberties. GAO, Technology Assessment: Using Biometrics for Border Security,GAO-03-174 (Washington, D.C.: Nov. 15, 2002). Examples from GAO TA reports wereincluded here given our familiarity with GAO products; numerous non-GAO examples ofresearch objectives and questions exist.Page 7GAO-20-246G Technology Assessment Handbook
Chapter 1: The Importance of TechnologyAssessment Designpolicymakers more broadly, and are not addressed to a specific federalagency or entity. 31.2 Importance ofSpending Time onDesignDeveloping a written TA design helps TA teams agree on andcommunicate a clear plan of action to the project team and the team’sadvisers, requesters, and other stakeholders. Written TA designs alsohelp guide and coordinate the project team’s activities and facilitatedocumentation of decisions and procedures in the final report. In addition,focusing the TA on answering specific researchable questions can assistteams to define and select the appropriate scope, approach, and type ofproduct, ensuring usefulness of the product to the intended users. Morespecific reasons for spending time on systematically designing a TAinclude: Enhance its quality, credibility, and usefulness Ensure independence of the analysis Ensure effective use of resources, including timeData collection and quality assurance of data can
contain additional information and details related to TA design, such as elaborating on specific methodologies that could be applied within this general design framework, including those designed to identify policy options. Below is a summary of the approach we used to identify and document TA design steps and considerations for this handbook.
1.2 Importance of Spending Time on Design 9 Chapter 2 Technology Assessment Scope and Design 11 2.1 Sound Technology Assessment Design 11 2.2 Stages and Considerations for Technology Assessment Design 12 2.2.1 GAO Technology Assessment Design Examples 18 Chapter 3 Approaches to Select Technology Assessment Design and Implementation Challenges .
assessment. In addition, several other educational assessment terms are defined: diagnostic assessment, curriculum-embedded assessment, universal screening assessment, and progress-monitoring assessment. I. FORMATIVE ASSESSMENT . The FAST SCASS definition of formative assessment developed in 2006 is “Formative assessment is a process used
2b. Content of GCSE (9-1) in Design and Technology (J310) 8 2c. Content of Principles of Design and Technology (J310/01) 10 2d. Non-exam assessment content - Iterative Design Challenge (J310/02, 03) 21 2e. Prior knowledge, learning and progression 26 3 Assessment of GCSE (9-1) in Design and Technology 27 3a. Forms of assessment 27 3b.
Information Technology Security . Risk Assessment Procedures . Supersedes Handbook OCIO-07 “Handbook for Information Technology Security Risk Assessment Procedures” dated 05/12/2003. For technical questions relating to this handbook, please contact Jennifer Beale on
PPAT Assessment Candidate and Educator Handbook 5 This Candidate and Educator Handbook will help you understand the history and background of the PPAT assessment and will provide you with a general overview as well as the specific details needed to submit the contents of your performance assessment. The PPAT assessment has been developed by a team of exemplary national educators.
Handbook for Curriculum Assessment Curriculum Assessment: An Overview What is curriculum assessment? Curriculum assessment is a process of gathering and analyzing information from multiple sources in order to improve student learning in sustainable ways. Why bother assessing curriculum? Curriculum assessment can serve several major purposes:
Youth handbooks are the Lion Cub Handbook, Tiger Handbook, Wolf Handbook, Bear Handbook, and Webelos Handbook. An assortment of handbooks are available for adult leaders, including: Den Leader Guides Cub Scout Leader Handbook Cub Scout Leader How-To Handbook Visit www.scoutstuff.org or the Bloomington Scout Shop for more
FAA-H-8083-3 Airplane Flying Handbook FAA-H-8083-6 Advanced Avionics Handbook FAA-H-8083-9 Aviation Instructor's Handbook FAA-H-8083-15 Instrument Flying Handbook FAA-H-8083-16 Instrument Procedures Handbook FAA-H-8083-25 Pilot's Handbook of Aeronautical Knowledge FAA-H-8083-30 Aviation Maintenance Technician Handbook— General FAA-H-8083 .
