key points. During the flight today, I carefully read the first two chapters and found a lot of interesting content in it.
machine learning models
In the past decade, many remarkable machine learning models have been contributed by industry, with impacts far exceeding those of academia. This is mainly because, compared to academia, industry has unparalleled advantages in data acquisition, computing resources, and financial support.
At the national level, the United States has made the most significant contributions in this field, followed by Europe and China.
If we look at the model's parameter size, in most cases, models with more training data tend to have more parameters. Similarly, models with more parameters often perform better.
When discussing computational resources, we usually measure them in PetaFLOPs (quadrillions of floating-point operations).
Overview of foundational models
When focusing on foundational models, it is worth noting that most of these models remain open-source.
Note: The relationship between Foundation Models and traditional machine learning models lies in the fact that foundation models provide a powerful pre-trained platform for building task-specific machine learning models. By leveraging the pre-training capabilities of these models, we can more effectively train and deploy new machine learning models without having to conduct large-scale data training from scratch.
In the past year, Google has taken the lead in contributing foundation models, followed by Meta. In academia, the University of California, Berkeley (UC Berkeley) has made the most significant contributions.
From a national perspective, the United States is leading in the development and application of foundation models, with China following closely behind.
Training cost
The part that the general public is most interested in is the cost involved in playing with large models!
First, let's estimate the training cost.
It can be seen that the training cost of AI models is directly related to their computational requirements.
With the continuous increase in training costs, this growth has effectively excluded traditional artificial intelligence research centers such as universities, which are unable to independently develop cutting-edge foundation models. For example, President Biden's AI executive order aims to balance the gap between industry and academia by creating a national AI research resource, providing non-industry researchers with the necessary computing power and data resources to conduct more advanced AI research.
It's somewhat similar to the Apollo Moon landing program in history. 🐶