#144 From AlexNet to Gemini Ultra: The Evolution of AI Compute Requirements
Fresh & Hot curated AI happenings in one snack. Never miss a byte 🍔
This snack byte will take approx 3 minutes to consume.
AI BYTE # 📢: From AlexNet to Gemini Ultra: The Evolution of AI Compute Requirements
The term “compute” in AI models denotes the computational resources required to train and operate a machine learning model.
This includes both hardware components such as
Central Processing Units (CPUs),
Graphics Processing Units (GPUs),
Tensor Processing Units (TPUs),
Field-Programmable Gate Arrays (FPGAs),
Computational tasks like floating-point operations (FLOPs) and
Parallel computing.
The complexity of the model and the size of the training dataset are two primary factors that directly influence the amount of compute needed.
More complex models and larger training datasets necessitate greater computational resources for training.
For instance, the computational requirements of Transformer models increase quadratically with the length of the input sequence.
In recent years, there has been an exponential increase in the compute usage of notable AI models. This trend has been particularly pronounced over the last five years.
Since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time. This metric has grown by more than 300,000x since 2012.
This rapid rise in compute demand has significant implications. Models that require more computation often have larger environmental footprints.
Training advanced AI models consumes a large amount of electricity, mostly from fossil fuels, increasing greenhouse gas emissions. Moreover, AI systems need enormous amounts of fresh water to cool their processors and generate electricity.
Furthermore, companies typically have more access to computational resources than academic institutions.
As tech companies have invested vast amounts of money into acquiring computational resources and datasets, researchers in academia and the public sector have been left behind.
AlexNet, a seminal paper that popularized the practice of using GPUs to enhance AI models, required an estimated 470 petaFLOPs for training.
The original Transformer, released in 2017, required around 7,400 petaFLOPs.
Google’s Gemini Ultra, one of the current state-of-the-art foundation models, required 50 billion petaFLOPs.
However, it’s important to note that Gemini Ultra is still in its development stage and requires highly advanced infrastructure for operations.