The day before yesterday, Stanford University released the "Artificial Intelligence Index Report 2024".
Click the following link to view the full report: https://aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_AI-Index-Report-2024.pdf
The following is a brief summary of the report. For more details, please download the report from the above link.
Report Overview
Welcome to the seventh edition of the Artificial Intelligence Index Report. The 2024 report is our most comprehensive version to date, released at a moment when the impact of AI on society has never been more critical. This year, we have expanded our scope of research to cover a broader range of key trends, including advancements in AI technology, public perceptions of this technology, and the geopolitical dynamics surrounding its development. This edition contains more raw data than ever before, with new estimates of AI training costs, a detailed analysis of the landscape of responsible AI, and an entirely new chapter focused on the impact of AI on science and medicine. The AI Index Report aggregates, refines, and visualizes data related to artificial intelligence. Our mission is to provide rigorously vetted, widely sourced objective data so that policymakers, researchers, corporate executives, journalists, and the public can understand the complex field of artificial intelligence more comprehensively and intricately. The index is globally recognized as one of the most credible and authoritative sources of AI data and insights. Previous editions have been cited in major newspapers such as The New York Times, Bloomberg, and The Guardian, garnered hundreds of academic citations, and have been referenced by senior policymakers in the U.S., UK, EU, and other regions. This year's edition surpasses all previous versions in scale and scope, reflecting the growing importance of AI in all of our lives.
Top 10 Takeaways
: Artificial intelligence has surpassed humans in certain tasks, such as image classification, visual reasoning, and English understanding. However, in more complex tasks like competition-level mathematics, visual common-sense reasoning, and planning, AI still lags behind. : In 2023, the industry produced 51 significant machine learning models, while academia contributed only 15. Additionally, the number of significant models co-produced by industry and academia reached 21, setting a new record. ：According to the AI Index, the training cost of the most advanced AI models has reached an unprecedented level. For example, the training cost of OpenAI's GPT-4 is estimated at $78 million, while the computational cost for Google's Gemini Ultra is $191 million. ：In 2023, 61 significant AI models originated from U.S.-based institutions, far exceeding the EU’s 21 and China’s 15. ：Recent research indicates a significant lack of standardization in responsible AI reports. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks, a practice that complicates systematic comparisons of the risks and limitations of top-tier AI models. : Despite an overall decline in private AI investment last year, funding for generative AI surged, nearly increasing eightfold since 2022 to reach $25.2 billion. Major players including OpenAI, Anthropic, Hugging Face, and Inflection reported large fundraising rounds. : Multiple studies in 2023 evaluated the impact of AI on the workforce, showing that AI helped workers complete tasks faster and improved work quality. These studies also demonstrated AI's potential to narrow the skill gap between low-skilled and high-skilled workers. However, other research warned that using AI without proper oversight could lead to performance degradation. ：In 2022, AI began driving scientific discoveries. By 2023, more significant AI applications related to science emerged — from AlphaDev, which improves the efficiency of algorithm sorting, to GNoME, which promotes material discovery. ：In the past year and over the past five years, there has been a noticeable increase in the number of AI-related regulations in the United States. In 2023, the number of AI-related regulations reached 25, a significant rise from just one regulation in 2016. In just the last year, the total number of AI-related regulations increased by 56.3%. ：A survey by Ipsos showed that the proportion of people who believe AI will greatly affect their lives in the next three to five years increased from 60% to 66% in the past year. Additionally, 52% expressed concerns about AI products and services, up 13 percentage points from 2022. In the U.S., data from Pew indicates that 52% of Americans feel more concern than excitement about AI, up from 37% in 2022.
Content by chapters
This report is divided into 9 chapters. The content of each chapter:
Chapter One: Research and Development
: In 2023, the industry launched 51 significant machine learning models, while academia contributed 15. Additionally, the number of significant models produced through collaboration between industry and academia reached 21, setting a new record. : In 2023, a total of 149 foundation models were released, more than double the number in 2022. Among these newly released models, 65.7% were open source, compared to 44.4% in 2022 and 33.3% in 2021. : According to estimates by the AI Index, the training cost of the most advanced AI models has reached unprecedented levels. For example, the estimated training cost for OpenAI's GPT-4 is $78 million, while the computational cost for Google's Gemini Ultra is $191 million. ：In 2023, 61 significant AI models originated from U.S. institutions, far exceeding the EU's 21 and China's 15. ：From 2021 to 2022, the global number of AI patent grants increased by 62.7%. Since 2010, the number of AI patents granted has increased by more than 31 times. ：In 2022, China led the world in AI patent origins with a 61.1% share, significantly ahead of the U.S.'s 20.9%. Since 2010, the U.S. share in the AI patent field has declined from 54.1%. : Since 2011, the number of AI-related projects on GitHub has been continuously increasing, growing from 845 in 2011 to approximately 1.8 million in 2023. In particular, in 2023, the total number of AI projects on GitHub increased by 59.3%. At the same time, the total number of stars for these projects on GitHub also significantly increased in 2023, rising from 4 million in 2022 to 12.2 million. : From 2010 to 2022, the total number of AI-related publications nearly tripled, increasing from about 88,000 to over 240,000. In the past year, this growth rate was 1.1%.
Chapter Two: Technical Performance
AI has surpassed human performance in multiple benchmarks, including image classification, visual reasoning, and English comprehension. However, it still lags behind in more complex tasks such as competition-level mathematics, visual common-sense reasoning, and planning. Traditional AI systems usually have single functions; for example, language models excel at text understanding but perform poorly in image processing, and vice versa. However, recent advances have led to the development of powerful multimodal models, such as Google's Gemini and OpenAI's GPT-4. These models demonstrate flexibility, capable of handling images and text, and in some cases even audio. . As AI models reach performance saturation on established benchmarks such as ImageNet, SQuAD, and SuperGLUE, researchers have introduced more challenging new benchmarks. New benchmarks that emerged in 2023 include SWE-bench for coding, HEIM for image generation, MMMU for general reasoning, MoCa for moral reasoning, AgentBench for agent-based behavior, and HaluEval for hallucination. . New AI models like SegmentAnything and Skoltech are used to generate specialized data for tasks such as image segmentation and 3D reconstruction. Data is key to improving AI technology. Using AI to create more data enhances current capabilities and paves the way for future algorithmic improvements, especially in more challenging tasks. . As generative models produce high-quality text, images, etc., benchmarking has gradually shifted toward including human evaluations, such as the Chatbot Arena leaderboard, rather than purely computational rankings like ImageNet or SQuAD. Public perception of AI is becoming an increasingly important factor in tracking AI progress. . The integration of language modeling with robotics has led to more flexible robot systems, such as PaLM-E and RT-2. These models not only enhance the capabilities of robots but also enable them to ask questions, marking an important step toward robots that can interact more effectively with the real world. . For a long time, creating AI agent systems capable of autonomous operation within specific environments has been a challenge for computer scientists. However, emerging studies indicate that the performance of autonomous AI agents is improving. Current agents can master complex games like Minecraft and effectively handle real-world tasks such as online shopping and research assistance. In ten carefully selected AI benchmarks, the closed model outperforms the open model with a median performance advantage of 24.2%. The difference in performance between closed and open models has important implications for the AI policy debate.
Chapter Three: Responsible AI
The latest research shows significant deficiencies in standardization in responsible AI reports. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks, making it complex to systematically compare the risks and limitations of top-tier AI models. Political deepfake videos have already influenced elections around the world, and recent studies show that the accuracy of existing AI deepfake video methods varies. Moreover, new projects like CounterCloud demonstrate how AI can easily create and spread false content. . Previously, red teaming for AI models focused mainly on testing adversarial prompts that are intuitively meaningful to humans. This year, researchers uncovered some less obvious strategies that can cause LLMs to exhibit harmful behaviors, such as making the model endlessly repeat random words. . A global survey on responsible AI highlights that companies are most concerned about AI-related issues including privacy, data security, and reliability. The survey shows that organizations are beginning to take steps to mitigate these risks. However, globally, most companies have so far addressed only a small portion of these risks. . Multiple studies indicate that the generative outputs of popular LLMs may contain copyrighted materials, such as excerpts from The New York Times or movie scenes. Whether this kind of output constitutes copyright infringement is becoming a central legal issue. . The newly introduced Foundation Model Transparency Index shows that AI developers lack transparency, particularly in disclosing training data and methodologies. This closed approach hinders efforts to further understand the robustness and safety of AI systems. . In the past year, there has been extensive discussion between AI scholars and practitioners regarding the focus on immediate model risks (such as algorithmic discrimination) versus potential long-term existential threats. It is challenging to distinguish claims grounded in scientific evidence from those that should inform policy-making, a challenge exacerbated by the specificity of short-term risks and the theoretical nature of long-term existential threats. . According to the AI Incident Database, which tracks events related to AI misuse, 123 incidents were reported in 2023, an increase of 32.3 percentage points compared to 2022. Since 2013, AI incidents have grown more than twentyfold. A notable example includes AI-generated deepfake videos of Taylor Swift with sexual innuendo that were widely shared online. . Researchers found that ChatGPT leans toward the Democratic Party in the U.S. and the Labour Party in the UK. This discovery has raised concerns about the tool's potential to influence users' political views, especially in a year marked by significant global elections.
Chapter Four: Economy
. Despite an overall decline in private AI investments last year, funding for generative AI surged, nearly increasing eightfold from 2022 to reach $25.2 billion. Major companies including OpenAI, Anthropic, Hugging Face, and Inflection reported large-scale fundraising rounds. . In 2023, the total amount of AI investment in the U.S. reached $67.2 billion, almost 8.7 times that of China, the country with the second-highest investment. Meanwhile, since 2022, private AI investments in China and the EU (including the UK) have decreased by 44.2% and 14.1%, respectively, while U.S. investments grew by 22.1%. In 2022, AI-related job postings accounted for 2.0% of all US job postings, a figure that dropped to 1.6% in 2023. The decrease in AI job listings is attributed to fewer postings from leading AI companies and a declining proportion of technical roles within these companies. A new McKinsey survey finds that 42% of surveyed organizations report cost reductions after implementing AI (including generative AI), while 59% report increased revenues. There was a 10-percentage-point increase in respondents reporting cost reductions compared to last year, indicating that AI is driving significant business efficiency improvements. Global private AI investment has declined for a second consecutive year, though the decline is less steep than the sharp drop from 2021 to 2022. The number of newly funded AI companies surged to 1,812, an increase of 40.6% over last year. The 2023 McKinsey report reveals that now 55% of organizations use AI (including generative AI) in at least one business unit or function, up from 50% in 2022 and 20% in 2017. Since overtaking Japan in 2013 to become the world's largest installer of industrial robots, China has significantly widened the gap with its closest competitor. In 2013, China accounted for 20.8% of global installations, and by 2022, this share had risen to 52.4%. . In 2017, collaborative robots accounted for only 2.8% of all new industrial robot installations, a figure that rose to 9.9% by 2022. Similarly, in 2022, the number of service robot installations increased across all application categories outside of medical robots. This trend not only indicates an increase in the total number of robot installations but also reflects a growing emphasis on deploying robots in human-facing service roles. . Multiple studies in 2023 evaluated the impact of AI on the workforce, showing that AI helps workers complete tasks faster and improves work quality. These studies also demonstrated AI's potential to bridge the skill gap between low-skilled and high-skilled workers. However, other research warns that using AI without proper regulation may lead to performance degradation. In 2023, AI was mentioned in 394 earnings calls (nearly 80% of all Fortune 500 companies), a significant increase from 266 mentions in 2022. Since 2018, the number of times AI has been mentioned in earnings calls of Fortune 500 companies has nearly doubled. Among these, the most frequently mentioned topic was generative AI, accounting for 19.7% of all calls.
Chapter Five: Science and Medicine
In 2022, AI began driving scientific discoveries. In 2023, more significant AI applications related to science emerged—from AlphaDev, which improves algorithm sorting efficiency, to GNoME, which facilitates material discovery. In 2023, several important medical systems were launched, including EVEscape, which enhances epidemic prediction capabilities, and AlphaMissence, which assists in AI-driven mutation classification. AI is increasingly being used to drive medical advancements. In recent years, AI systems have shown significant improvements on the MedQA benchmark, a key test for evaluating AI clinical knowledge. In 2023, the outstanding model GPT-4 Medprompt achieved an accuracy rate of 90.2%, an increase of 22.6 percentage points over the highest score in 2022. Since the launch of this benchmark in 2019, AI performance on MedQA has nearly tripled. In 2022, the FDA approved 139 AI-related medical devices, an increase of 12.1% compared to 2021. Since 2012, the number of AI-related medical devices approved by the FDA has increased by more than 45 times. AI is increasingly being used for practical medical purposes.
Chapter Six: Education
Although the number of undergraduate graduates in computer science in the U.S. and Canada has continued to grow over the past decade, the number of students choosing to pursue graduate degrees has leveled off. Since 2018, there has been a slight decline in the number of graduates with master’s and doctoral degrees in computer science. In 2011, the proportion of new AI doctoral students choosing to enter industry (40.9%) and academia (41.6%) was roughly equal. However, by 2022, the majority (70.7%) of doctoral students joined the industrial sector after graduation, while only 20.0% chose academia. In just the past year, the proportion of AI doctoral students entering industry increased by 5.3 percentage points, indicating an intensification of talent loss from universities to the industrial sector. In 2019, 13% of new AI professors in the U.S. and Canada came from industry. By 2021, this figure dropped to 11%, and further decreased to 7% in 2022. This trend indicates a gradual reduction in the migration of high-level AI talent from industry to academia. In 2022, the proportion of international graduates in computer science at the undergraduate, master's, and doctoral levels decreased compared to 2021, with a particularly noticeable decline in the number of international students in the master's category. In 2022, 201,000 AP Computer Science exams were taken. The number of students taking these exams has increased more than tenfold since 2007. However, recent evidence suggests that students in large high schools and suburban schools are more likely to have access to computer science courses. Since 2017, the number of AI-related higher education degree programs taught in English has tripled, with steady growth each year over the past five years. Universities around the world are offering more degree programs focused on AI. The UK and Germany lead Europe in the production of bachelor's, master's, and doctoral graduates in informatics, computer science, computer engineering, and information technology. On a per capita basis, Finland leads in the production of bachelor's and doctoral graduates, while Ireland leads in the production of master's graduates.
Chapter Seven: Policy and Governance
. Over the past year, and within the last five years, there has been a notable rise in the number of AI-related regulations in the United States. In 2023, the number of AI-related regulations reached 25, a significant increase from just one regulation in 2016. In just the last year alone, the total number of AI-related regulations grew by 56.3%. . In 2023, policymakers on both sides of the Atlantic proposed major initiatives to advance AI regulation. The European Union reached an agreement on the terms of the AI Act, which is set to become landmark legislation in 2024. Meanwhile, President Biden signed an executive order on AI, marking the most notable AI policy initiative in the U.S. that year. In 2023, there was a notable increase in AI-related legislative proposals at the federal level, with the number of bills introduced rising from 88 in 2022 to 181 in 2023, more than doubling. Mentions of AI in global legislative proceedings nearly doubled, increasing from 1,247 in 2022 to 2,175 in 2023. In 2023, a total of 49 countries mentioned AI in their legislative processes. Additionally, in 2023, at least one country on each continent discussed AI, highlighting the global impact of AI policy discourse. The number of U.S. regulatory agencies issuing AI regulations in 2023 increased from 17 in 2022 to 21, indicating growing attention from U.S. regulators towards AI regulations. New regulatory agencies that formulated AI-related regulations for the first time in 2023 include the Department of Transportation, the Department of Energy, and the Occupational Safety and Health Administration.
Chapter Eight: Diversity
While white students still make up the largest proportion of new resident graduates at the bachelor's, master's, and doctoral levels, the representation of other racial groups, such as Asians, Hispanics, and Black or African American students, is increasing. For example, since 2011, the proportion of Asian bachelor's degree graduates in computer science has increased by 19.8 percentage points, while the proportion of Hispanic bachelor's degree graduates has grown by 5.2 percentage points. Every European country surveyed reported more male than female graduates in bachelor’s, master’s, and doctoral programs in informatics, computer science, computer engineering, and information technology. Although the gender gap has narrowed in most countries over the past decade, the pace of narrowing is slow. The proportion of female students taking the AP Computer Science exam increased from 16.8% in 2007 to 30.5% in 2022. Similarly, the participation of Asian, Hispanic/Latino, and Black/African American students in AP Computer Science has steadily increased year by year.
Chapter Nine: Public Opinion
According to a survey by Ipsos, the proportion of people who believed that AI would significantly impact their lives in the next three to five years increased from 60% to 66% over the past year. Additionally, 52% expressed concerns about AI products and services, which was a rise of 13 percentage points compared to 2022. In the United States, Pew data showed that 52% of Americans were more worried than excited about AI, up from 38% in 2022. In 2022, several developed Western countries, including Germany, the Netherlands, Australia, Belgium, Canada, and the United States, had the least positive attitudes toward AI products and services. Since then, every one of these countries has seen an increase in the proportion of respondents recognizing the benefits of AI, with the most significant change occurring in the Netherlands. In an Ipsos survey, only 37% of respondents believed that AI would improve their jobs. Only 34% expected AI to boost the economy, and 32% thought it would improve the job market. There are significant demographic differences in the perception of AI's potential to enhance living standards, with younger generations generally being more optimistic. For example, 59% of Generation Z respondents believe AI will improve entertainment options, compared to only 40% of Baby Boomers. Additionally, individuals with higher income and education levels are more optimistic about the positive impacts of AI on entertainment, health, and the economy, compared to their lower-income and less-educated peers. An international survey from the University of Toronto shows that 63% of respondents are aware of ChatGPT. Among those who are aware, about half report using ChatGPT at least once a week.