Have LLMs Hit a Wall? Microsoft Chief Satya Nadella Tackled This Hot-Button Issue at Microsoft Ignite 2024, Offering a Refreshingly Candid Take on the Discussion

"There's a lot of debate on whether we have hit the wall with scaling laws. Is it going to continue? The thing to remember, at the end of the day, is that these are not physical laws."

Microsoft Ignite 2024 saw Microsoft chief Satya Nadella weigh in on the hot-button issue of whether LLMs have hit a wall.

“There’s a lot of debate on whether we have hit the wall with scaling laws. Is it going to continue? The thing to remember, at the end of the day, is that these are not physical laws. They are just empirical observations that held true, much like how Moore’s Law did for a long time,” he said.

Nadella welcomed the skepticism and debates, calling them beneficial to push innovation in areas such as model architectures, data regimes, and systems architecture. He also discussed OpenAI’s new scaling law, which focuses on test-time computing, and how it will be integrated into features like Copilot Think Deeper, powered by OpenAI’s o1.

In a recent earnings call, NVIDIA chief Jensen Huang said that OpenAI o1 had introduced a new scaling law called ‘test-time scaling’, which consumed a lot of computing resources. Microsoft is working closely with NVIDIA to scale test-time computing for OpenAI.

Nadella emphasized the importance of maximizing value in the most efficient way. “Last month, we introduced new clusters with H200s that became available. We’re very excited about it,” said Nadella. He added that with their stack optimization between H100 and H200, Azure can deliver performance for everything from inference to training.

Efficiency Wars: Tokens, Watts, and Dollars

“Tokens per watt plus dollar is the best way to think about the new currency of performance,” said Nadella, adding that Microsoft will continue to build new data center intelligence factories.

Nadella introduced a new metric that reflects the efficiency of generating tokens, considering both energy consumption (measured in watts) and cost (measured in dollars). This means that for every unit of energy (watt) used and every dollar spent, a certain number of tokens are produced.

Despite the progress, NVIDIA has yet to solve the inferencing challenge. Acknowledging the difficulties involved, Huang shared that their goal is to produce tokens at low latency.

“Inference is super hard. And the reason…is that you need the accuracy to be high…You need the throughput to be high so that the cost can be as low as possible. But you also need the latency to be low. And computers that are high-throughput and have latency are incredibly hard to build,” he said.

“Our hopes and dreams are that, someday, the world will do a ton of inference,” said Huang, adding that there will be thousands of AI-native start-ups that will generate tokens.

Microsoft also announced the preview of NVIDIA Blackwell AI infrastructure on Azure.

“Blackwell is