探索基礎推理模型的隱藏狀態，以降低推理效率低下

人工智能係統在模擬人類風格的推理，尤其是數學和邏輯方面取得了長足的進步。這些模型不僅會產生答案，還可以瀏覽一系列邏輯步驟以得出結論，從而提供了有關如何以及為什麼產生這些答案的見解。這種逐步推理，通常稱為經過思考鏈（COT），在機器如何處理複雜的解決問題的任務方面變得至關重要。

Artificial intelligence systems have made remarkable progress in simulating human-style reasoning, especially in domains like mathematics and logic. Unlike typical generative models, these systems generate a series of intermediate steps to reach a final answer, offering insights into the reasoning process. This step-by-step reasoning, often called Chain-of-Thought (CoT), is crucial for machines to handle complex problem-solving tasks.

人工智能係統在模擬人類風格的推理方面取得了顯著進步，尤其是在數學和邏輯等領域。與典型的生成模型不同，這些系統生成了一系列中間步驟以達到最終答案，從而提供了對推理過程的見解。這種逐步推理，通常稱為經過思考鏈（COT），對於計算機處理複雜的解決問題的任務至關重要。

A common challenge researchers face is the models' inefficiency during inference. The reasoning models may continue processing even after attaining a correct conclusion, leading to overthinking. This generates unnecessary tokens, increasing computational cost.

研究人員面臨的一個普遍挑戰是模型在推斷過程中的效率低下。推理模型即使得出正確的結論也可能會繼續處理，從而導致過度思考。這會產生不必要的令牌，從而增加了計算成本。

Many current approaches measure a model's confidence using verbal prompts or by analyzing multiple outputs. These "black-box" strategies ask the model to report how sure it is of its answer. However, they are often imprecise and computationally expensive. On the other hand, "white-box" methods investigate models' internal hidden states to extract signals that may correlate with answer correctness.

當前許多方法使用口頭提示或分析多個輸出來衡量模型的置信度。這些“黑框”策略要求該模型報告其答案的確定性。但是，它們通常不精確且計算昂貴。另一方面，“白框”方法研究了模型的內部隱藏狀態，以提取可能與答案正確性相關的信號。

Prior work has shown that a model's internal states can indeed indicate the validity of final answers. However, applying this to intermediate steps in long reasoning chains is still an underexplored direction.

先前的工作表明，模型的內部狀態確實可以表明最終答案的有效性。但是，將其應用於長期推理鏈中的中間步驟仍然是一個毫無疑問的方向。

To bridge this gap, a team of researchers from New York University and NYU Shanghai designed a lightweight probe—a simple two-layer neural network—to inspect a model's hidden states at intermediate reasoning steps. Their models of choice were the DeepSeek-R1-Distill series and QwQ-32B, known for their excellent step-by-step reasoning capabilities, tested across various datasets including AIME, GSM8K, and MATH. The researchers trained their probe to read the internal state associated with each chunk of reasoning and predict whether the current intermediate answer was correct.

為了彌合這一差距，來自紐約大學和紐約大學上海的一組研究人員設計了一個輕量級的探測器（一個簡單的兩層神經網絡），以檢查模型在中間推理步驟中的隱藏狀態。他們選擇的模型是DeepSeek-R1-Distill系列和QWQ-32B，以出色的逐步推理功能而聞名，這些功能在包括AIME，GSM8K和MATH在內的各種數據集中進行了測試。研究人員訓練了他們的探測，以閱讀與推理的每一部分相關的內部狀態，並預測當前的中間答案是否正確。

To construct their approach, they segmented each long CoT output into smaller parts or chunks, using markers like "wait" or "verify" to identify breaks in reasoning. They used the last token's hidden state in each chunk as a representation and matched this to a correctness label, which was judged using another model. These representations were then used to train the probe on binary classification tasks. The probe was fine-tuned using grid search across hyperparameters like learning rate and hidden layer size, with most models converging to linear probes—highlighting that correctness information is often linearly embedded in the hidden states.

為了構建他們的方法，他們使用“等待”或“驗證”等標記來識別推理中的斷裂，將每個長床輸出分為較小的部分或塊。他們將最後一個令牌在每個塊中的隱藏狀態用作表示形式，並將其與正確的標籤匹配，該標籤是使用另一個模型來判斷的。然後使用這些表示形式來訓練二進制分類任務的探測器。使用網格搜索跨越超參數（如學習率和隱藏層的大小）進行了微調，大多數模型都會收斂到線性探針 - 高燈表明，正確性信息通常是線性嵌入在隱藏狀態中的。

The probe worked effectively for fully formed answers and even showed the ability to predict correctness before an answer was completed, alluding to look-ahead capabilities.

該探測器在完全形成的答案中有效地起作用，甚至顯示了在答案完成之前預測正確性的能力，從而暗示了看起來很容易的功能。

Performance results were clear and quantifiable. The probes achieved ROC-AUC scores exceeding 0.9 for some datasets like AIME when using models like R1-Distill-Qwen-32B. Expected Calibration Errors (ECE) remained under 0.1, showcasing high reliability. For instance, R1-Distill-Qwen-32B had an ECE of just 0.01 on GSM8K and 0.06 on MATH.

性能結果清晰可量化。對於使用R1-Distill-Qwen-32b之類的模型時，對於某些數據集（例如AIME），探針的ROC-AUC得分超過了0.9。預期校準誤差（ECE）保持在0.1之下，顯示了高可靠性。例如，R1-DISTILL-QWEN-32B的ECE在GSM8K上僅為0.01，數學上的ECE為0.06。

In application, the probe was used to implement a confidence-based early exit strategy during inference. The reasoning process was halted when the probe's confidence in an answer exceeded a threshold. At a confidence threshold of 0.85, the accuracy remained at 88.2%, while the inference token count was reduced by 24%. Even at a threshold of 0.9, accuracy stayed at 88.6%, with a 19% token reduction. Compared to static exit methods, this dynamic strategy achieved up to 5% higher accuracy using the same or fewer tokens.

在應用中，探測器在推斷期間被用來實施基於置信的早期退出策略。當探測器對答案的信心超過閾值時，推理過程停止了。在0.85的置信度閾值下，準確性保持在88.2％，而推斷令牌計數降低了24％。即使在0.9的閾值下，準確性仍保持在88.6％，降低了令牌的19％。與靜態退出方法相比，使用相同或更少的令牌，這種動態策略的精度高達5％。

This study provides an efficient, integrated way for reasoning models to self-verify during inference. The researchers' approach highlights a gap—models inherently know when they're right, but they don't act on it. This research opens up avenues for smarter, more efficient reasoning systems by leveraging internal representations through probing. It demonstrates that tapping into what the model already "knows" can lead to significant improvements in both performance and resource use.

這項研究為推理模型在推理期間進行自我驗證提供了一種有效的綜合方法。研究人員的方法突出了一個差距 - 模型本質上知道它們何時正確，但他們不採取行動。這項研究通過探測利用內部表示，開闢了更智能，更有效的推理系統的途徑。它表明，利用已經“知道”的模型可以導致性能和資源使用的重大改進。