自回報的語言模型沒有註定：經過思考鏈的促進和細心推理查詢如何阻止不一致

META的首席AI科學家，現代AI的先驅之一Yann Lecun最近認為自回歸的大型語言模型（LLMS）從根本上存在缺陷。

Yann LeCun, Chief AI Scientist at Meta and one of the pioneers of modern AI, recently shared his thoughts on a fundamental limitation he sees in autoregressive Large Language Models (LLMs). According to LeCun, the probability of generating a correct response decreases exponentially with each token, making them impractical for long-form, reliable AI interactions.

META的首席AI科學家，現代AI的先驅之一Yann Lecun最近分享了他對他在自回歸大型語言模型（LLMS）中看到的基本限制的想法。根據Lecun的說法，產生正確響應的概率隨著每個令牌的指數呈指數降低，使其對於長形式，可靠的AI相互作用而言是不切實際的。

While I deeply respect LeCun’s work and approach to AI development and resonate with many of his insights, I believe this particular claim overlooks some key aspects of how LLMs function in practice. In this post, I’ll explain why autoregressive models are not inherently divergent and doomed, and how techniques like Chain-of-Thought (CoT) and Attentive Reasoning Queries (ARQs)—a method we’ve developed to achieve high-accuracy customer interactions with Parlant—effectively prove otherwise.

儘管我深深地尊重Lecun的工作和AI發展的方法並與他的許多見解產生共鳴，但我相信這一特殊主張忽略了LLM在實踐中的功能的一些關鍵方面。在這篇文章中，我將解釋為什麼自回歸模型本質上沒有固有的分歧和注定，以及如何採用思想鏈（COT）和細心的推理查詢（ARQ），這是我們開發的一種方法來實現高準確的客戶與Parlant的相互作用 - 有效地證明了這一點。

What is Autoregression?

什麼是自動性？

At its core, an LLM is a probabilistic model trained to generate text one token at a time. Given an input context, the model predicts the most likely next token, feeds it back into the original sequence, and repeats the process iteratively until a stop condition is met. This allows the model to generate anything from short responses to entire articles.

LLM的核心是一種概率模型，該模型訓練了一次生成文本一個令牌。給定輸入上下文，該模型可以預測最有可能的令牌，將其饋回原始序列，然後迭代地重複該過程，直到滿足停止條件為止。這使該模型可以生成從簡短響應到整個文章的任何內容。

For a deeper dive into autoregression, check out our recent technical blog post.

要更深入地了解自動性，請查看我們最近的技術博客文章。

Do Generation Errors Compound Exponentially?

生成錯誤是否呈指數級化？

LeCun’s argument can be unpacked as follows:

Lecun的論點可以被解開，如下所示：

Let P be the probability of making a generation error at each token.

令P為在每個令牌上造成生成誤差的概率。

For an output of length n, the probability of maintaining coherence is (1-E)^n.

對於長度為n的輸出，保持相干性的概率為（1-e）^n。

This leads to LeCun’s conclusion that for sufficiently long responses, the likelihood of maintaining coherence exponentially approaches zero, suggesting that autoregressive LLMs are inherently flawed.

這得出了萊肯的結論，即對於足夠長的響應，保持連貫性的可能性將指數成倍接近零，這表明自回調的LLM固有有缺陷。

But here’s the problem: E is not constant.

但這是問題：e不是恆定的。

To put it simply, LeCun’s argument assumes that the probability of making a mistake in each new token is independent. However, LLMs don’t work that way.

簡而言之，Lecun的論點假設在每個新令牌中犯錯的可能性是獨立的。但是，LLM不起作用。

As an analogy to what allows LLMs to overcome this problem, imagine you’re telling a story: if you make a mistake in one sentence, you can still correct it in the next one to keep the narrative coherent. The same applies to LLMs, especially when techniques like Chain-of-Thought (CoT) prompting guide them toward better reasoning by helping them reassess their own outputs along the way.

為了類比使LLM可以克服這個問題的原因，想像一下您是在講一個故事：如果您在一個句子中犯了一個錯誤，那麼您仍然可以在下一個句子中糾正它以保持敘事相干。 LLM同樣適用，尤其是當諸如《經營鏈》（COT）之類的技術提示他們通過幫助他們在此過程中重新評估自己的輸出來指導他們更好地推理時。

Why This Assumption is Flawed

為什麼這個假設有缺陷

LLMs exhibit self-correction properties that prevent them from spiraling into incoherence.

LLM具有自校正特性，可防止它們螺旋形成不連貫。

Take Chain-of-Thought (CoT) prompting, which encourages the model to generate intermediate reasoning steps. CoT allows the model to consider multiple perspectives, improving its ability to converge to an acceptable answer. Similarly, Chain-of-Verification (CoV) and structured feedback mechanisms like ARQs guide the model in reinforcing valid outputs and discarding erroneous ones.

採取思想鏈（COT）提示，這鼓勵模型生成中間的推理步驟。 COT允許該模型考慮多種觀點，從而提高其收斂到可接受的答案的能力。同樣，驗證鏈（COV）和結構化反饋機制（如ARQ）指導該模型加強有效的輸出並丟棄錯誤的輸出。

A small mistake early on in the generation process doesn’t necessarily doom the final answer. Figuratively speaking, an LLM can double-check its work, backtrack, and correct errors on the go.

一代過程中的一個小錯誤不一定會注定最終答案。象徵性地說，LLM可以仔細檢查其工作，回溯和正確的錯誤。

Attentive Reasoning Queries (ARQs) are a Game-Changer

細心的推理查詢（ARQ）是改變遊戲規則的

At Parlant, we’ve taken this principle further in our work on Attentive Reasoning Queries (a research paper describing our results is currently in the works, but the implementation pattern can be explored in our open-source codebase). ARQs introduce reasoning blueprints that help the model maintain coherence throughout long completions by dynamically refocusing attention on key instructions at strategic points in the completion process, continuously preventing LLMs from diverging into incoherence. Using them, we’ve been able to maintain a large test suite that exhibits close to 100% consistency in generating correct completions for complex tasks.

在Parlant，我們在專門推理查詢的工作中進一步採取了這一原則（描述我們結果的研究論文目前正在起作用，但是可以在我們的開源代碼庫中探索實施模式）。 ARQ引入了推理藍圖，通過在完成過程中的戰略點上的關鍵指令進行動態重新關注，從而幫助該模型保持一致性，從而不斷防止LLMS分流為不一致。使用它們，我們已經能夠維護一個大型測試套件，該套件在為複雜的任務生成正確的完成時表現出接近100％的一致性。

This technique allows us to achieve much higher accuracy in AI-driven reasoning and instruction-following, which has been critical for us in enabling reliable and aligned customer-facing applications.

這項技術使我們能夠在AI驅動的推理和遵循指導範圍內實現更高的準確性，這對我們實現可靠且面向面向客戶的應用程序至關重要。

Autoregressive Models Are Here to Stay

自回歸模型在這裡留下來

We think autoregressive LLMs are far from doomed. While long-form coherence is a challenge, assuming an exponentially compounding error rate ignores key mechanisms that mitigate divergence—from Chain-of-Thought reasoning to structured reasoning like ARQs.

我們認為自回歸的LLM遠未註定。雖然長形式的連貫性是一個挑戰，但假設有指數級的複雜錯誤率忽略了減輕差異的關鍵機制，從想像力的推理到諸如ARQ之類的結構化推理。

If you’re interested in AI alignment and increasing the accuracy of chat agents using LLMs, feel free to explore Parlant’s open-source effort. Let’s continue refining how LLMs generate and structure knowledge.

如果您對使用LLM的AI對齊感興趣並提高聊天代理的準確性，請隨時探索Parlant的開源工作。讓我們繼續完善LLM的產生和結構知識。