ETSI GR SAI 004 V1.1

ETSI GR SAI 004 V1.1.1 (2020-12)GROUP REPORTSecuring Artificial Intelligence (SAI);Problem StatementDisclaimerThe present document has been produced and approved by the Secure AI (SAI) ETSI Industry Specification Group (ISG) andrepresents the views of those members who participated in this ISG.It does not necessarily represent the views of the entire ETSI membership.

2ETSI GR SAI 004 V1.1.1 (2020-12)ReferenceDGR/SAI-004Keywordsartificial intelligence, securityETSI650 Route des LuciolesF-06921 Sophia Antipolis Cedex - FRANCETel.: 33 4 92 94 42 00 Fax: 33 4 93 65 47 16Siret N 348 623 562 00017 - NAF 742 CAssociation à but non lucratif enregistrée à laSous-Préfecture de Grasse (06) N 7803/88Important noticeThe present document can be downloaded from:http://www.etsi.org/standards-searchThe present document may be made available in electronic versions and/or in print. The content of any electronic and/orprint versions of the present document shall not be modified without the prior written authorization of ETSI. In case of anyexisting or perceived difference in contents between such versions and/or in print, the prevailing version of an ETSIdeliverable is the one made publicly available in PDF format at www.etsi.org/deliver.Users of the present document should be aware that the document may be subject to revision or change of status.Information on the current status of this and other ETSI documents is available .aspxIf you find errors in the present document, please send your comment to one of the following pportStaff.aspxCopyright NotificationNo part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopyingand microfilm except as authorized by written permission of ETSI.The content of the PDF version shall not be modified without the written authorization of ETSI.The copyright and the foregoing restriction extend to reproduction in all media. ETSI 2020.All rights reserved.DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.3GPP and LTE are trademarks of ETSI registered for the benefit of its Members andof the 3GPP Organizational Partners.oneM2M logo is a trademark of ETSI registered for the benefit of its Members andof the oneM2M Partners.GSM and the GSM logo are trademarks registered and owned by the GSM Association.ETSI

3ETSI GR SAI 004 V1.1.1 (2020-12)ContentsIntellectual Property Rights .5Foreword.5Modal verbs terminology.51Scope .62References 5Normative references . 6Informative references . 6Definition of terms, symbols and abbreviations .8Terms. 8Symbols . 9Abbreviations . 9Context .9History . 9AI and machine learning . 10Data processing chain (machine learning). 10Overview . 10Data Acquisition . 12Description . 12Integrity challenges . 12Data Curation . 12Description . 12Integrity challenges . 12Model Design. 12Software Build . 12Training . 12Description . 12Confidentiality challenges . 13Integrity challenges . 13Availability challenges . 13Testing . 14Description . 14Availability challenges . 14Deployment and Inference . 14Description . 14Confidentiality challenges . 14Integrity challenges . 15Availability challenges . 15Upgrades . 15Description . 15Integrity challenges . 15Availability challenges . 15Design challenges and unintentional factors .15Introduction . 15Bias . 15Ethics . 16Introduction. 16Ethics and security challenges . 16Access to data. 16Decision-making . 17Obscurity . 17Summary . 17Ethics guidelines . 18Explainability . 18Software and hardware . 19ETSI

466.16.26.36.4ETSI GR SAI 004 V1.1.1 (2020-12)Attack types .19Poisoning . 19Input attack and evasion . 19Backdoor Attacks . 19Reverse Engineering. 207Misuse of AI .208Real world use cases and attacks .208.18.28.38.48.58.68.7Overview . 20Ad-blocker attacks . 21Malware Obfuscation . 21Deepfakes . 21Handwriting reproduction . 21Human voice . 21Fake conversation . 22Annex A:Bibliography .23History .24ETSI

5ETSI GR SAI 004 V1.1.1 (2020-12)Intellectual Property RightsEssential patentsIPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The informationpertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be foundin ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI inrespect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Webserver (https://ipr.etsi.org/).Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guaranteecan be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Webserver) which are, or may be, or may become, essential to the present document.TrademarksThe present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys noright to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document doesnot constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.ForewordThis Group Report (GR) has been produced by ETSI Industry Specification Group (ISG) Secure AI (SAI).Modal verbs terminologyIn the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to beinterpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions)."must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.ETSI

61ETSI GR SAI 004 V1.1.1 (2020-12)ScopeThe present document describes the problem of securing AI-based systems and solutions, with a focus on machinelearning, and the challenges relating to confidentiality, integrity and availability at each stage of the machine learninglifecycle. It also describes some of the broader challenges of AI systems including bias, ethics and explainability. Anumber of different attack vectors are described, as well as several real-world use cases and attacks.2References2.1Normative referencesNormative references are not applicable in the present document.2.2Informative referencesReferences are either specific (identified by date of publication and/or edition number or version number) ornon-specific. For specific references, only the cited version applies. For non-specific references, the latest version of thereferenced document (including any amendments) applies.NOTE:While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guaranteetheir long term validity.The following referenced documents are not necessary for the application of the present document but they assist theuser with regard to a particular subject area.[i.1]NOTE:[i.2]NOTE:[i.3]NOTE:Florian Tramèr, Pascal Dupré, Gili Rusak, Giancarlo Pellegrino, Dan Boneh: "AdVersarial:Perceptual Ad Blocking meets Adversarial Machine Learning", In Proceedings of the 2019, ACMSIGSAC Conference on Computer and Communications Security Pages 2005-2021November t Millar, Niall McLaughlin, Jesus Martinez del Rincon, Paul Miller, Ziming Zhao:"DANdroid: A Multi-View Discriminative Adversarial Network for Obfuscated Android MalwareDetection" in Proceedings of the 10th ACM Conference on Data and Application Security andPrivacy e, D. : "Understanding artificial intelligence ethics and safety: A guide for the responsibledesign and implementation of AI systems in the public sector", The Alan Turing Institute ]High Level Expert Group on Artificial Intelligence, European Commission: "Ethics Guidelines forTrustworthy AI", April 2019.[i.5]UK Department for Digital, Culture, Media & Sport: "Data Ethics Framework", August 2018.[i.6]Song, C., Ristenpart, T., and Shmatikov, V.: "Machine Learning Models that Remember TooMuch", ACM CCS 17, Dallas, TX, USA.[i.7]"Model-Agnostic Meta-Learning for Fast Adaptation of Deep 3.03400.pdf."Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning".https://arxiv.org/abs/1712.05526.ETSI

7[i.9]NOTE:[i.10]NOTE:ETSI GR SAI 004 V1.1.1 (2020-12)Tom S. F. Haines, Oisin Mac Aodha, and Gabriel J. Brostow. 2016: "My Text in YourHandwriting", ACM Trans. Graph. 35, 3, Article 26 (June 2016), 18 pages.https://doi.org/10.1145/2886099.K. Eykholt et al.: "Robust Physical-World Attacks on Deep Learning Visual Classification", 2018IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018,pp. [i.11]Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart, 2016: "Stealingmachine learning models via prediction APIs", In Proceedings of the 25th USENIX Conference onSecurity Symposium (SEC"16). USENIX Association, USA, 601-618.[i.12]Seong Joon Oh, Max Augustin, Bernt Schiele, Mario Fritz: "Towards reverse-engineering blackbox neural networks Max-Planck Institute for Informatics", Saarland Informatics Campus,Saarbrucken, Germany Published as a conference paper at ICLR 2018.[i.13]Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves,Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu WaveNet: "A Generative Model for RawAudio", September abs/1609.03499.Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, AllanDafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C.Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, Simon Beard, Haydn Belfield,Sebastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson,Roman Yampolskiy, Dario Amodei: "The Malicious Use of Artifici

ETSI 2 ETSI GR SAI 004 V1.1.1 (2020-12) Reference DGR/SAI-004 Keywords artificial intelligence, security ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE