AUDIT AND ASSURANCE AUDITING ARTIFICIAL INTELLIGENCE
A U D I TA N DA S S U R A N C EAUDITINGARTIFICI ALINTELLIGENCE
2AUDITING ARTIFICIAL INTELLIGENCECONTENTS44Potential Impact of Artificial Intelligenceon OrganizationsWhy Should Auditors Care About AI? 4 / Challenges for the Auditor 6 / Mapping COBIT to Strategy: A Visual Representation of How to Apply COBIT 2019 in the Auditing of AI 8 / Challenges and Solutions for the AI Auditor9Conclusion12Acknowledgments10Resources and References for Auditing AI 2018 ISACA. All Rights Reserved.
3AUDITING ARTIFICIAL INTELLIGENCEABSTRACTThere are many potential challenges for IT auditors preparing to equip themselves toaudit artificial intelligence (AI). But solutions do exist that can transform challenges intosuccesses. This white paper focuses on what auditors need to know as they prepare tofocus on AI. It explores the definition of AI, describes the challenges of auditing AI, anddiscusses how the current version of COBIT (COBIT 2019) can be leveraged to audit AI.Additionally, it identifies other frameworks that are also relevant today. Auditors willexplore initial keys to successfully auditing AI and uncover relevant references. 2018 ISACA. All Rights Reserved.
4AUDITING ARTIFICIAL INTELLIGENCEPotential Impact of ArtificialIntelligence on OrganizationsThere are many truths and half-truths out theresignificant impact on many areas in the business worldindustries and professions. Some industries have adoptedimpact on the audit profession as well, given auditors’concerning the impact that AI will have across a range ofelements of the technology faster than others, withvarying degrees of success and challenges. Given thehype surrounding AI we can be certain that there will be afor the foreseeable future. AI will have a far-reachingneed to provide assurance around it. The purpose of thispaper is to prepare auditors for what to expect and how toapproach AI in a real-world audit scenario.Why Should Auditors Care About AI?Like many complex emerging technologies, AI is definedrepresented in the smaller circles. Machine learning is a subsetresearch team has elected to retain flexibility in thedata and learn for themselves, changing their algorithms asin many ways, by many experts. Whereas the ISACAof AI in that it focuses on machines’ ability to receive a set ofdefinition, due to the technology’s ever-changing scopeand context, one general definition can serve as anindicator as to the nature and purpose of AI. Russell andNorvig, two of the leading minds in the field, call AI thestudy of “intelligent agents,” devices that perceive theirenvironment and take actions that maximize their chanceof successfully achieving their goals.1 AI may be envisioned as a large circle with several smallercircles within it. AI, which is machines carrying out tasks basedon algorithms in an “intelligent” manner,2 is the large circle;2other, more specific types of AI, such as machine learning, are1223344 processing.33AI does not operate based on a set of predetermined rules.Predetermined rules are associated with traditional softwareengineering. However, an excessive number of rules tends toinhibit the technology’s ability to learn and adapt to itscircumstances. Therefore, AI does not always operate based1Two other concepts may also be helpful in understandingAI:needed as they learn more about the information they areon a predefined set of rules.Challenges for the AuditorTractica Research expects AI software revenue to growfrom US 3.2 billion in 2016 to US 89.9 billion by 2025.44With the support of adjacent technologies (such as cloudcomputing and storage), AI has emerged from the so-called “AI winter” of 2010 to garner up to US 40 billion ofRussell, S.; P. Norvig; Artificial Intelligence: A Modern Approach (3rd Edition), Pearson, USA, 2009, h-3rd-Edition/PGM156683.htmlVenkatesan, M.; “Artificial Intelligence vs. Machine Learning vs. Deep Learning,” Data Science Central, 7 May -vs-deep-learningIbid.Tractica Research, “Artificial Intelligence Software Market to Reach 89.8 Billion in Annual Worldwide Revenue by 2025,” 21 December 25 2018 ISACA. All Rights Reserved.
5AUDITING ARTIFICIAL INTELLIGENCEinvestment capital, at the same time productiondeployments have been limited.5source of audit challenges. However, this assumes that5AI’s rise has been accompanied by the traditional lag timebetween early adoption and the establishment of regulatoryand compliance frameworks. There is, for example, nomature auditing framework in place detailing AIsubprocesses, nor are there any AI-specific regulations,standards or mandates. Clark pioneered the cross-industryprocess for data mining (CRISP-DM) framework in early2018, but individual auditors are challenged with how toperform audits successfully when there are virtually nowidely adopted precedents for handling AI use cases.66In addition to a lack of explicit audit standards around AI,there are additional challenges impacting the auditprocess. As previously noted, the definition of AI isfrequently debated and the IT world, including auditors,has not reached a common definition or taxonomy onwhich to specify a set of world-class practices.Moreover, AI systems and solutions vary widely from eachother, and the vast set of existing and emergingtechnologies foundational to AI architecture give birth tocomplex systems. This complexity points to a highlikelihood of uncertainty around the scope of AI within thebusiness. Despite this uncertainty in the business,auditors are fairly well positioned to take on theirresponsibilities relative to AI. Good technology auditorsare already likely to possess enough skill andunderstanding to effectively assess AI in the enterprise.In addition, the complexity of AI and the shortage ofqualified data scientists will routinely lead to theoutsourcing of AI development projects to one or morethird-party resources. A coherent understanding ofenterprise AI will be dispersed—and, over time, perhapseven lost—across tiers of AI providers. This willsubsequently increase the challenge for the AI auditor.While there will undoubtedly be challenges for AI auditorsas they ramp up for their new responsibilities, the situationis not as dire as might be assumed. The “black box” effectoften ascribed to machine learning is often cited as a5566traditional technology auditors are responsible forauditing algorithms. This is not the case. IT auditorsshould look at the governance of AI and the integrationamong systems. Although the algorithms should beaudited by model specialists, auditors having a basicunderstanding of the would be beneficial. In fact, auditorsalready do so, using information in regulations such as USOffice of the Comptroller of the Currency (OCC) 2011-12.There are also claims the challenge is due to a lack ofacademic research and industry publications on the topic.This, too, is inaccurate. There is a considerable amount ofresearch, but it is highly technical and not typically aimedat the traditional auditor. Historically, traditional IT auditorshave looked at governance and integration, without divingdeeply into algorithms.Most enterprises have not yet begun to think about how AImay play a role in their businesses, so they are unlikely tohave a documented plan to align AI use cases to thebusiness or to recognize return on AI investments.However, if they do decide to adopt AI, executives willdemand clarity of a higher order as they begin their effortsto develop an effective AI strategy. Because the businesscase and strategy documents represent typical startingpoints on the AI journey, auditors will be challenged tocascade down the COBIT 2019 hierarchy from thestrategic to the tactical parts of the audit.In sum, IT auditors should not go down the path ofoverthinking the challenges of auditing AI. Reflecting onhow they first audited cloud computing or cybersecurityshould provide them with a useful frame of reference. Forexample, it is unlikely they examined all the protocols indepth and tested that the Open Systems Interconnection(OSI) layer 5 implementation was functioningappropriately. Instead, with AI, as with those previous newtechnologies, auditors will focus on the controls andgovernance structures that are in place and determinethat they are operating effectively. Auditors can providesome assurance by focusing on the business and ITgovernance aspects.Bughin, J.; E. Hazan; S. Ramaswamy; M. Chui; T. Allas; P. Dahlstrom; N. Henke; M. Trench; “Artificial Intelligence: The Next Digital Frontier?” McKinsey Global Institute, June 2017,https://www.mckinsey.com/ k, A.; “The Machine Learning Audit—CRISP-DM Framework,” ISACA Journal, vol. 1, 2018, work.aspx 2018 ISACA. All Rights Reserved.
6AUDITING ARTIFICIAL INTELLIGENCEMapping COBIT to Strategy: AVisual Representation of Howto Apply COBIT 2019 in theAuditing of AIAs the application of AI in the business world is still in itsearly stages, there is limited guidance on how to approachauditing an AI initiative for an organization. Therefore, thisexample leverages ISACA’s COBIT 2019 framework as aareas of risk should then be compiled in a document suchas a risk and control matrix (RCM), which lists each riskand related controls. COBIT 2019 provides a goodframework for considering the risk of any initiative orprocess within an organization. There are several examples of risk related to AI strategy: starting point. The COBIT 2019 framework provides the outcomes, base practices and work products across auditor with tools—including process descriptions, desiredvirtually all the IT domains—to enable the auditor toprovide assurance over the AI initiative for anyorganization.A starting point for an audit of an organization’s AI is todefine the scope and objectives of the audit and considerrisk to the organization related to the AI initiative. These Lack of alignment between IT plans and business needs IT plans that are inconsistent with the organization’sexpectations or requirements Improper translation of IT tactical plans from the IT strategicplans Ineffective governance structures that fail to ensureaccountability and responsibility for IT processes related to theAI functionFigure 1 highlights several examples of processes withinCOBIT 2019 that may provide help in compiling a list ofrisks and controls for the AI initiative within anorganization. FIGURE 1: Select COBIT 2019 Governance and Management Objectives Relevant to AI Risk and Controls ReviewEDM01—EnsuredGovernanceFramework Settingand MaintenanceEDM02—EnsuredBenefits �EnsuredRisk M01 Ensured GovernanceFramework Setting and MaintenanceEDM01.02 Direct the governance system.EDM01.03 Monitor the governance system.APO02 Managed StrategyAPO01—ManagedI&T tionAPO05—ManagedPortfolioAPO06—ManagedBudget and CostsAPO07—ManagedHuman olutionsIdentificationand BuildBAI04—ManagedAvailabilityand �ManagedIT ChangesBAI07—ManagedIT ChangeAcceptance agedProjectsDSS02—ManagedService Requestsand tinuityAPO02.03 Define target digital capabilities.APO02.04 Conduct a gap analysis.APO02.05 Define the strategic plan androad map.DSS01—ManagedOperationsMEA02—ManagedSystem of InternalControlMEA03—ManagedCompliance WithExternalRequirementsAPO04 Managed InnovationAPO04.04 Assess the potential of emergingtechnologies and innovative ideas.APO04.06 Monitor the implementation anduse of innovation.MEA01—ManagedPerformance vicesSource: ISACA, COBIT 2019 Framework: Introduction and Methodology, USA, 2018 2018 ISACA. All Rights Reserved.DSS06—ManagedBusinessProcess ControlsMEA04—ManagedAssurance
7AUDITING ARTIFICIAL INTELLIGENCEDSS06 provides a more in-depth example of how theauditor can leverage COBIT 2019 during the course of anAI assurance review.DSS06 Managed Business Process Controls includesmanagement practice DSS06.05 Ensure traceability andaccountability for information events, which could be usedto ensure AI activity audit trails provide sufficientFIGURE 2: COBIT 2019: Relevance of DSS06 to AI information to understand the rationale behind every AIdecision made within the organization. The DSS06.05description (figure 2) follows: “Ensure that businessinformation can be traced to an originating business eventand associated with accountable parties. Thisdiscoverability provides assurance that businessinformation is reliable and has been processed inaccordance with defined objectives.”77A. Component: Process (cont.)Management PracticeExample MetricsDSS06.05 Ensure traceability and accountability for information events. a. Number of incidents in which transaction history cannot be recoveredEnsure that business information can be traced to an originatingb. Percent of completeness of traceable transaction logbusiness event and associated with accountable parties. Thisdiscoverability provides assurance that business information is reliableand has been processed in accordance with defined objectives.ActivitiesCapability Level1. Capture source information, supporting evidence and the record of transactions.22. Define retention requirements, based on business requirements, to meet operational, financial reporting and compliance needs.33. Dispose of source information, supporting evidence and the record of transactions in accordance with the retention policy.Related Guidance (Standards, Frameworks, Compliance Requirements)Detailed ReferenceNo related guidance for this management practiceSource: ISACA, COBIT 2019 Framework: Governance and Management Objectives, USA, 2018Process outcomes in COBIT 2019 are derived from the1“Business information is traced to an originated business2practice itself, and for DSS06.05, can be articulated asevent and is associated with accountable parties.”The following activities are listed for DSS06.05:3Capture source information, supporting evidence and the recordof transactions.Define retention requirements, based on business requirements,to meet operational, financial reporting and compliance needs.Dispose of source information, supporting evidence and therecord of transactions in accordance with the retention policy.Figure 3 shows the inputs and outputs from DSS06.05.77ISACA, COBIT 2019 Framework: Governance and Management Objectives, USA, 2018, orkGovernance-and-Management-Objectives.aspx 2018 ISACA. All Rights Reserved.
8AUDITING ARTIFICIAL INTELLIGENCEFIGURE 3: COBIT 2019: DSS06 Inputs and OutputsC. Component: Information Flows and Items (see also Section 3.6) (cont.)Management PracticeInputsDSS06.04 Manage errors and exceptions.FromDescriptionDSS06.05 Ensure traceability and accountability forinformation events.DSS06.06 Secure information assets.OutputsDescriptionToError reports and rootcause analysisInternalEvidence of errorcorrection andremediationMEA02.04Record of transactionsInternalRetention requirementsInternal;APO14.09Reports of violationsDSS05.03Related Guidance (Standards, Frameworks, Compliance Requirements)Detailed ReferenceNational Institute of Standards and Technology Special Publication800-37, Revision 2, September 20173.1 Preparation (Task 10, 11): Inputs and OutputsSource: ISACA, COBIT 2019 Framework: Governance and Management Objectives, USA, 2018Audits should evaluate the work products, retentionrequirements and records of transaction as part offieldwork testing. Criteria the auditor would use for testinginclude, “Does the decision made by AI seem appropriate,given the decision inputs and use case?”Challenges and Solutions forthe AI AuditorWhile there are several potential challenges for IT auditorspreparing to equip themselves to audit AI, solutions doexist that can convert the challenges into successes. Thelist in figure 4 provides examples.FIGURE 4: Challenges and Solutions for AI AuditingCHALLENGES FOR THE AUDITOR OF AI1. Adopt and adapt existing frameworks and regulations.3. Uncertain definitions and taxonomies of AI3. Explain and communicate proactively about AI with stakeholders.2. Limited precedents for AI use cases4. Wide variance among AI systems and solutions5. Become informed about AI design and architecture to set properscope.6. Lack of explicit AI auditing guidance6. Focus on transparency through an iterative process. Focus oncontrols and governance, not algorithms.7. Lack of strategic starting points7. Involve all stakeholders.8. Possibly steep learning curve for the AI auditor8. Become informed about AI design and engage specialists asneeded.9. Supplier risk created by AI outsourcing to third partiesThe following information expands on the keys to successlisted in figure 4, to help auditors address the challengesof auditing AI: Become informed about AI design and architecture to setproper scope. AI includes a large set of technologies, peopleand processes and, therefore, will require significant attention tocontrols, policies and governance. AI architecture may combine8Op cit Tractica2. Explain and communicate proactively about AI with stakeholders.4. Become informed about AI design and architecture to set properscope.5. Emerging nature of AI technology8KEYS TO THE SUCCESSFUL AUDITING OF AI1. Immature auditing frameworks or regulations specific to AI9. Document architectural practices for cross-team transparency.programming, data warehousing, stream processing platforms,machine learning tool kits, algorithms, cloud computing, cloudstorage, computing clusters, compute kernels, applicationsoftware testing and debugging, data process and modeling,and commercial off-the-shelf (COTS) software. From a skillsperspective, AI projects may require data scientists, dataengineers, data architects and programmers capable in Python,R, Java and matrix laboratory (MATLAB).8 2018 ISACA. All Rights Reserved.8
9 AUDITING ARTIFICIAL INTELLIGENCE Involve all stakeholders. AI not only integrates a variety ofimpediment to a successful audit. COBIT 2019 and otherteams and external third parties. Internal stakeholders involveexisting AI use cases that will be encountered in the field. Also,business leaders engaged with the AI strategy. The use of cloudUnited States Health Insurance Portability and Accountabilityexisting frameworks can be adopted to handle most of theenterprise technologies but also involves multiple internalfrom a regulatory perspective, existing charters such as theengineering and security teams on the technical side andAct (HIPAA) and Fair Lending Act and the European Union’scomputing is widespread with AI and implies that third partiesGeneral Data Protection Regulation (GDPR) can be adopted towill control part of the infrastructure. Where cloud computing isprovide legal guidance. The existing frameworks and regulationused, for example, auditors must address risk (such as vendor lock-in and partitioned knowledge) differently from t
2 AUDITING ARTIFICIAL INTELLIGENCE CONTENTS 4 Potential Impact of Artificial Intelligence on Organizations 4 Why Should Auditors Care About AI? 4 / Challenges for the Auditor 6 / Mapping COBIT to Strategy: A Visual Representation of How to Apply COBIT 2019 in the Auditing of AI 8 / Challenges and Solutions for the AI Auditor 9 Conclusion 10 Resources and References for Auditing AI 12 .
of Auditing and Assurance-Introduction (Auditing 1) and Auditing and Assurance-Intermediate (Auditing 2). This course is designed to provide an introduction to auditing and assurance services. Level of Proficiency in Auditing 1: Foundation Subject Learning Outcome Upon completion of the subj
Introduction to Assurance and Financial Statement Auditing 1 Chapter 1 An Introduction to Assurance and Financial Statement Auditing 2 Tips for Learning Auditing 4 The Demand for Auditing and Assurance 5 Principals and Agents 5 The Role of Auditing 6 An Assurance Analogy: The Case of
auditing, performance auditing, comprehensive auditing, internal auditing and forensic auditing, as well as providing assurance on subject matter other than historical financial information. Major chapter sections The framework for assurance engagements and the types of assurance engagements
Chapter 05 - Auditing and Advanced Threat Analytics 1h 28m Topic A: Configuring Auditing for Windows Server 2016 Overview of Auditing The Purpose of Auditing Types of Events Auditing Goals Auditing File and Object Access Demo - Configuring Auditing Topic B: Advanced Auditing and Management Advanced Auditing
SECTION-1 (AUDITING) INTRODUCTION TO AUDITING STRUCTURE: 1.1 Objectives 1.2 Introduction -an overview of auditing 1.3 Origin and evolution 1.4 Definition 1.5 Salient features 1.6 Scope of auditing 1.7 Principles of auditing 1.8 Objects of audit 1.9 Detection and prevention of fraud 1.2 1.10 Concept of " true and fair view"
INTRODUCTION TO AUDITING 1 - 37 Introduction 2 Meaning and Nature of Auditing 4 Relationship among Book-keeping, Accountancy and Auditing 6 Objectives of Audit 8 Subsidiary Object of an Audit 10 Types of Audit 15 Standard on Auditing 26. UNIT - II. AUDITOR AND EXECUTION OF AUDIT 38 - 89 PART (A): APPOINTMENT, QUALIFICATION AND DISQUALIFICATION
Auditing and Assurance Services, 17e (Arens/Elder/Beasley) Chapter 1 The Demand for Audit and Other Assurance Services 1.1 Learning Objective 1-1 1) In the auditing process, A) the types and amounts of evidence remain constant from audit to audit. B) the criteria for evaluating information will not vary depending on the information being audited.
Baking equipment Electrical equipment Other equipment Furniture Cooking equipment and starter packs : 37 Annual costs for cooking skills lessons: o Summary of annual costs for cooking skills lessons : o Store cupboard ingredients breakdown and costs o Ingredient breakdown and costs for suggested recipes o Other consumables : o : Replacing equipm nt: 41 Cooking and food skills for .
