CHAPTER 1: Research & Development - Stanford University

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N Saccording to the world average. For example, an FWCI of0.75 means 25% fewer citations than the world average.To assess how academic-corporate collaborationsimpact the Field-Weighted Citation Impact (FWCI) ofAI publications from different geographic regions,see Figure 1.1.6. FWCI measures how the number ofcitations received by publications compares with theaverage number of citations received by other similarpublications in the same year, discipline, and format(book, article, conference paper, etc.). A value of 1.0represents the world average. More than or less than 1means publications are cited more or less than expected,The chart shows the FWCI for all peer-reviewed AIpublications on the y-axis and the total number (on a logscale) of academic-corporate co-authored publicationson the x-axis. To increase the signal-to-noise ratio of theFWCI metric, only countries that have more than 1,000peer-reviewed AI publications in 2020 are included.PEER-REVIEWED AI PUBLICATIONS' FIELD-WEIGHTED CITATION IMPACT and NUMBER of ACADEMIC-CORPORATEPEER-REVIEWED AI PUBLICATIONS, 2019Peer-Reviewed AI Publications' Field-Weighted Citation Impact (FWCI)Source: Elsevier/Scopus, 2020 Chart: 2021 AI Index Report3SingaporeHong ysiaItalySpainGermanySouth KoreaEuropean razilUnited StatesUnited KingdomIndia051020501002005001,0002,000Number of Academic-Corporate Peer-Reviewed AI Publications (Log Scale)Figure 1.1.6CHAPTER 1 PREVIEW11

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SA I J O U R N A L P U B L I CAT I O N SOverviewThe next three sections chart the trends in the publicationof AI journals, conference publications, and patents, aswell as their respective citations that provide a signalfor R&D impact, based on data from Microsoft AcademicGraph. MAG3 is a knowledge graph consisting of more than225 million publications (at the end of November 2019).Overall, the number of AI journal publications in 2020 is 5.4times higher than it was in 2000 (Figure 1.1.7a). In 2020, thenumber of AI journal publications increased by 34.5% from2019—a much higher percentage growth than from 2018 to2019 (19.6%). Similarly, the share of AI journal publicationsamong all publications in the world has jumped by 0.4percentage points in 2020, higher than the average of 0.03percentage points in the past five years (Figure 1.1.7b).NUMBER of AI JOURNAL PUBLICATIONS, 2000-20Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index ReportNumber of AI Journal Publications (in 000Figure 1.1.7aAI JOURNAL PUBLICATIONS (% of ALL JOURNAL PUBLICATIONS), 000AI Journal Publications (% of All Publications)Source: Microso Academic Graph, 2020 Chart: 2021 AI Index ReportFigure 1.1.7b3 See “An Overview of Microsoft Academic Service (MAS) and Applications” and “A Review of Microsoft Academic Services for Science of Science Studies” for more details.CHAPTER 1 PREVIEW12

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SBy RegionFigure 1.1.8 shows the share of AI journals—the dominantpublication entity in terms of numbers in the MAGdatabase—by region between 2000 and 2020. East Asia& Pacific, Europe & Central Asia, and North America areresponsible for the majority of AI journal publications inthe past 21 years, while the lead position among the threeregions changes over time. In 2020, East Asia & Pacific heldthe highest share (26.7%), followed by Europe & CentralAsia (13.3%) and North America (14.0%). Additionally,in the last 10 years, South Asia, and Middle East & NorthAfrica saw the most significant growth, as the number ofAI journal publications in those two regions grew six- andfourfold, respectively.AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index Report30%26.7%, East Asia & Pacific20%14.0%, North America13.3%, Europe & Central Asia10%4.9%, South 203.1%, Middle East & North Africa1.3%, Latin America & Caribbean0.3%, Sub-Saharan Africa2000AI Journal Publications (% of World Total)40%Figure 1.1.8CHAPTER 1 PREVIEW13

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SBy Geographic AreaFigure 1.1.9 shows that among the three major AI powers, China has had the largest share of AI journal publications in theworld since 2017, with 18.0% in 2020, followed by the United States (12.3%) and the European Union (8.6%).AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20AI Journal Publications (% of World Total)Source: Microso Academic Graph, 2020 Chart: 2021 AI Index Report25%20%18.0%, China15%12.3%, US10%8.6%, 092008200720062005200420032002200120000%Figure 1.1.9CitationIn terms of the highest share of AI journal citations, Figure 1.1.10 shows that China (20.7%) overtook the United States(19.8%) in 2020 for the first time, while the European Union continued to lose overall share.AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index ReportAI Journal Citations (% of World 062005200420032002200120000%Figure 1.1.10CHAPTER 1 PREVIEW14

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SA I C O N F E R E N C E P U B L I CAT I O N SOverviewBetween 2000 and 2019, the number of AI conference publications increased fourfold, although the growth flattenedout in the past ten years, with the number of publications in 2019 just 1.09 times higher than the number in 2010.4NUMBER of AI CONFERENCE PUBLICATIONS, 2000-20Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index ReportNumber of Publications (in 20000Figure 1.1.11aAI CONFERENCE PUBLICATIONS (% of ALL CONFERENCE PUBLICATIONS), 2000-20Source: Microso Academic Graph, 2020 Chart: 2021 AI Index ReportAI Conference Publications (% of All 20032002200120000%Figure 1.1.11b4 Note that conference data in 2020 on the MAG system is not yet complete. See the Appendix for details.CHAPTER 1 PREVIEW15

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SBy RegionAsia & Pacific took the lead starting in 2004, accountingfor more than 27% in 2020. North America overtookEurope & Central Asia to claim second place in 2018,accounting for 20.1%, followed by 21.7% in 2020.Figure 1.1.12 shows that, similar to the trends in AIjournal publication, East Asia & Pacific, Europe & CentralAsia, and North America are the world’s dominantsources for AI conference publications. Specifically, EastAI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by REGION, 2000-20Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index Report30%27.3%, East Asia & Pacific21.7%, North America20%18.6%, Europe & Central 001199920000%20215.1%, South Asia2.2%, Middle East & North Africa1.7%, Latin America & Caribbean0.3%, Sub-Saharan Africa2020AI Conference Publications (% of World Total)40%Figure 1.1.12CHAPTER 1 PREVIEW16

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SBy Geographic Areais almost nine times higher than it was in 2000. Theshare of conference publications for the European Unionpeaked in 2011 and continues to decline.China overtook the United States in the share of AIconference publications in the world in 2019 (Figure1.1.13). Its share has grown significantly since 2000.China’s percentage of AI conference publications in 2019AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-20AI Conference Publications (% of World Total)Source: Microso Academic Graph, 2020 Chart: 2021 AI Index Report25%20%19.4% US15%15.2% China12.8% 020092008200720062005200420032002200120000%Figure 1.1.13Citation21 years. The United States tops the list with 40.1% ofoverall citations in 2020, followed by China (11.8%) andthe European Union (10.9%).With respect to citations of AI conference publications,Figure 1.1.14 shows that the United States has held adominant lead among the major powers over the pastAI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2000-2040.1% US40%30%20%11.8% China10%10.9% 200820072006200520042003200220010%2000AI Conference Citations (% of World Total)Source: Microso Academic Graph, 2020 Chart: 2021 AI Index ReportFigure 1.1.14CHAPTER 1 PREVIEW17

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SA I PAT E N T SOverviewThe total number of AI patents published in the worldhas been steadily increasing in the past two decades,growing from 21,806 in 2000 to more than 4.5 times that,or 101,876, in 2019 (Figure 1.1.15a). The share of AI patentspublished in the world exhibits a lesser increase, fromNUMBER of AI PATENT PUBLICATIONS, 2000-20around 2% in 2000 to 2.9% in 2020 (Figure 1.1.15b). The AIpatent data is incomplete—only 8% of the dataset in 2020includes a country or regional affiliation. There is reasonto question the data on the share of AI patent publicationsby both region and geographic area, and it is therefore notincluded in the main report. See the Appendix for details.Number of AI Patent Publications (in Thousands)Source: Microsoft Academic Graph, 2020 Chart: 2021 AI Index 00Figure 1.1.15aAI PATENT PUBLICATIONS (% of ALL PATENT PUBLICATIONS), 000AI Patent Publications (% of All Publications)Source: Microso Academic Graph, 2020 Chart: 2021 AI Index ReportFigure 1.1.15bCHAPTER 1 PREVIEW18

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SA R X I V P U B L I CAT I O N SOverviewIn addition to the traditional avenues forpublishing academic papers (discussedabove), AI researchers have embraced thepractice of publishing their work (oftenpre–peer review) on arXiv, an onlinerepository of electronic preprints. arXivallows researchers to share their findingsbefore submitting them to journals andconferences, which greatly acceleratesthe cycle of information discovery anddissemination. The number of AI-relatedpublications in this section includespreprints on arXiv under cs.AI (artificialintelligence), cs.CL (computation andlanguage), cs.CV (computer vision), cs.NE(neural and evolutionary computing),cs.RO (robotics), cs.LG (machine learning incomputer science), and stat.ML (machinelearning in statistics).In just six years, the number of AI-related publications on arXiv grewmore than sixfold, from 5,478 in 2015 to 34,736 in 2020 (Figure 1.1.16).NUMBER of AI-RELATED PUBLICATIONS on ARXIV, 2015-20Number of AI-Related Publications on arXiv (in Thousands)Source: arXiv, 2020 Chart: 2021 AI Index Report30201002015201620172018By Region2019Figure 1.1.16The analysis by region shows that while North America still holds the lead in the global share of arXiV AI-relatedpublications, its share has been decreasing—from 41.6% in 2017 to 36.3% in 2020 (Figure 1.1.17). Meanwhile, the share ofpublications in East Asia & Pacific has grown steadily in the past five years—from 17.3% in 2015 to 26.5% in 2020.ARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by REGION, 2015-20AI-Related Publications on arXiv (% of World Total)Source: arXiv, 2020 Chart: 2021 AI Index Report40%36.3% North America30%26.5% East Asia & Paci c22.9% Europe & Central Asia20%10%4.0% South Asia2.5% Middle East & North Africa1.3% Latin America & Caribbean0.3% Sub-Saharan Africa0%2015CHAPTER 1 PREVIEW201620172018201920202021Figure 1.1.17192020

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SBy Geographic AreaWhile the total number of AI-related publications on arXivis increasing among the three major AI powers, China iscatching up with the United States (Figure 1.1.18a andFigure 1.1.18b). The share of publication counts by theEuropean Union, on the other hand, has remained largelyunchanged.NUMBER of AI-RELATED PUBLICATIONS on ARXIV by GEOGRAPHIC AREA, 2015-20Source: arXiv, 2020 Chart: 2021 AI Index Report11,280USNumber of AI-Related Publications on ,0000201520162017201820192020Figure 1.1.18aARXIV AI-RELATED PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2015-20AI-Related Publications on arXiv (% of World Total)Source: arXiv, 2020 Chart: 2021 AI Index Report32.5% US30%20%18.7% EU15.7% China10%0%201520162017201820192020Figure 1.1.18bCHAPTER 1 PREVIEW20

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 2021By Field of StudyAmong the six fields of study related to AI on arXiv,the number of publications in Robotics (cs.RO) andMachine Learning in computer science (cs.LG) haveseen the fastest growth between 2015 and 2020,increasing by 11 times and 10 times respectively(Figure 1.1.19). In 2020, cs.LG and Computer Vision(cs.CV) lead in the overall number of publications,accounting for 32.0% and 31.7%, respectively, of all AIrelated publications on arXiv. Between 2019 and 2020,the fastest-growing categories of the seven studiedhere were Computation and Language (cs.CL), by35.4%, and cs.RO, by 35.8%.1.1 P U B L I CAT I O N SAmong the six fields ofstudy related to AI on arXiv,the number of publicationsin Robotics (cs.RO) andMachine Learning incomputer science (cs.LG) have seen the fastestgrowth between 2015 and2020, increasing by 11 timesand 10 times respectively.NUMBER of AI-RELATED PUBLICATIONS on ARXIV by FIELD of STUDY 2015-20Source: arXiv, 2020 Chart: 2021 AI Index Report11,001 cs.CVNumber of AI-Related Publications on arXiv11,098 cs.LG10,0008,0006,0005,573 cs.CL4,0002,571 cs.RO2,0001,923 cs.AI1,818 stat.ML743 cs.NE0201520162017201820192020Figure 1.1.19CHAPTER 1 PREVIEW21

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211.1 P U B L I CAT I O N SDeep Learning Papers on arXivWith increased access to data and significantimprovements in computing power, the fieldof deep learning (DL) is growing at breakneckspeed. Researchers from Nesta used a topicmodeling algorithm to identify the deep learningpapers on arXiv by analyzing the abstract ofarXiv papers under the Computer Science (CS)and Machine Learning in Statistics (state.ML)categories. Figure 1.1.20 suggests that in thelast five years alone, the overall number of DLpublications on arXiv grew almost sixfold.NUMBER of DEEP LEARNING PUBLICATIONS on ARXIV, 2010-19Number of Deep Learning Publications on arXiv (in Thousands)Source: arXiv/Nesta, 2020 Chart: 2021 AI Index 20192020Figure 1.1.20CHAPTER 1 PREVIEW22

Artificial IntelligenceIndex Report 2021CHAPTER 1:RESEARCH &D E V E LO P M E N T1. 2 C O N F E R E N C E SConference attendance is an indication of broader industrial and academic interest in a scientific field. In the past 20 years,AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends inattendance at and submissions to major AI conferences.1.2 CONFERENCESC O N F E R E N C E AT T E N DA N C ELast year saw a significant increase in participation levelsat AI conferences, as most were offered through a virtualformat. Only the 34th Association for the Advancementof Artificial Intelligence (AAAI) Conference on ArtificialIntelligence was held in person in February 2020.Conference organizers report that a virtual format allowsfor higher attendance of researchers from all over theworld, though exact attendance numbers are difficult tomeasure.Due to the atypical nature of 2020 conference attendancedata, the 11 major AI conferences in 2019 have beensplit into two categories based on 2019 attendance data:large AI conferences with over 3,000 attendees and smallAI conferences with fewer than 3,000 attendees. Figure1.2.1 shows that in 2020, the total number of attendeesacross nine conferences almost doubled.5 In particular,the International Conference on Intelligent Robotsand Systems (IROS) extended the virtual conferenceto allow users to watch events for up to three months,which explains the high attendance count. Because theInternational Joint Conference on Artificial Intelligence(IJCAI) was held in 2019 and January 2021—but not in2020—it does not appear on the charts.Conference organizersreport that a virtualformat allows for higherattendance of researchersfrom all over the world,though exact attendancenumbers are difficult tomeasure.5 For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference; For the KR conference,the attendance in 2020 is based on the number of registrations; For the ICPAS conference, the attendance of 450 in 2020 is an estimate as some participants may have used anonymous Zoom accounts.CHAPTER 1 PREVIEW23

CHAPTER 1:RESEARCH &D E V E LO P M E N TArtificial IntelligenceIndex Report 20211. 2 C O N F E R E N C E SATTENDANCE at LARGE AI CONFERENCES, 2010-20Source: Conference Data Chart: 2021 AI Index Report25,719 IROS25,00022,011 NeurIPSNumber of Attendees20,00015,00010,800 ICML10,0007,500 CVPR5,0004,884 AAAI20203,015 0 ICRAFigure 1.2.1ATTENDANCE at SMALL AI CONFERENCES, 20

academia and industry in the United States has grown in importance and popularity, made visible by the proliferation of industry-university research centers as well as corporate contributions to university research. Figure 1.1.5 shows that between 2015 and 2019, the United States produced the highest number of hybrid