探索基础推理模型的隐藏状态，以降低推理效率低下

人工智能系统在模拟人类风格的推理，尤其是数学和逻辑方面取得了长足的进步。这些模型不仅会产生答案，还可以浏览一系列逻辑步骤以得出结论，从而提供了有关如何以及为什么产生这些答案的见解。这种逐步推理，通常称为经过思考链（COT），在机器如何处理复杂的解决问题的任务方面变得至关重要。

Artificial intelligence systems have made remarkable progress in simulating human-style reasoning, especially in domains like mathematics and logic. Unlike typical generative models, these systems generate a series of intermediate steps to reach a final answer, offering insights into the reasoning process. This step-by-step reasoning, often called Chain-of-Thought (CoT), is crucial for machines to handle complex problem-solving tasks.

人工智能系统在模拟人类风格的推理方面取得了显着进步，尤其是在数学和逻辑等领域。与典型的生成模型不同，这些系统生成了一系列中间步骤以达到最终答案，从而提供了对推理过程的见解。这种逐步推理，通常称为经过思考链（COT），对于计算机处理复杂的解决问题的任务至关重要。

A common challenge researchers face is the models' inefficiency during inference. The reasoning models may continue processing even after attaining a correct conclusion, leading to overthinking. This generates unnecessary tokens, increasing computational cost.

研究人员面临的一个普遍挑战是模型在推断过程中的效率低下。推理模型即使得出正确的结论也可能会继续处理，从而导致过度思考。这会产生不必要的令牌，从而增加了计算成本。

Many current approaches measure a model's confidence using verbal prompts or by analyzing multiple outputs. These "black-box" strategies ask the model to report how sure it is of its answer. However, they are often imprecise and computationally expensive. On the other hand, "white-box" methods investigate models' internal hidden states to extract signals that may correlate with answer correctness.

当前许多方法使用口头提示或分析多个输出来衡量模型的置信度。这些“黑框”策略要求该模型报告其答案的确定性。但是，它们通常不精确且计算昂贵。另一方面，“白框”方法研究了模型的内部隐藏状态，以提取可能与答案正确性相关的信号。

Prior work has shown that a model's internal states can indeed indicate the validity of final answers. However, applying this to intermediate steps in long reasoning chains is still an underexplored direction.

先前的工作表明，模型的内部状态确实可以表明最终答案的有效性。但是，将其应用于长期推理链中的中间步骤仍然是一个毫无疑问的方向。

To bridge this gap, a team of researchers from New York University and NYU Shanghai designed a lightweight probe—a simple two-layer neural network—to inspect a model's hidden states at intermediate reasoning steps. Their models of choice were the DeepSeek-R1-Distill series and QwQ-32B, known for their excellent step-by-step reasoning capabilities, tested across various datasets including AIME, GSM8K, and MATH. The researchers trained their probe to read the internal state associated with each chunk of reasoning and predict whether the current intermediate answer was correct.

为了弥合这一差距，来自纽约大学和纽约大学上海的一组研究人员设计了一个轻量级的探测器（一个简单的两层神经网络），以检查模型在中间推理步骤中的隐藏状态。他们选择的模型是DeepSeek-R1-Distill系列和QWQ-32B，以出色的逐步推理功能而闻名，这些功能在包括AIME，GSM8K和MATH在内的各种数据集中进行了测试。研究人员训练了他们的探测，以阅读与推理的每一部分相关的内部状态，并预测当前的中间答案是否正确。

To construct their approach, they segmented each long CoT output into smaller parts or chunks, using markers like "wait" or "verify" to identify breaks in reasoning. They used the last token's hidden state in each chunk as a representation and matched this to a correctness label, which was judged using another model. These representations were then used to train the probe on binary classification tasks. The probe was fine-tuned using grid search across hyperparameters like learning rate and hidden layer size, with most models converging to linear probes—highlighting that correctness information is often linearly embedded in the hidden states.

为了构建他们的方法，他们使用“等待”或“验证”等标记来识别推理中的断裂，将每个长床输出分为较小的部分或块。他们将最后一个令牌在每个块中的隐藏状态用作表示形式，并将其与正确的标签匹配，该标签是使用另一个模型来判断的。然后使用这些表示形式来训练二进制分类任务的探测器。使用网格搜索跨越超参数（如学习率和隐藏层的大小）进行了微调，大多数模型都会收敛到线性探针 - 高灯表明，正确性信息通常是线性嵌入在隐藏状态中的。

The probe worked effectively for fully formed answers and even showed the ability to predict correctness before an answer was completed, alluding to look-ahead capabilities.

该探测器在完全形成的答案中有效地起作用，甚至显示了在答案完成之前预测正确性的能力，从而暗示了看起来很容易的功能。

Performance results were clear and quantifiable. The probes achieved ROC-AUC scores exceeding 0.9 for some datasets like AIME when using models like R1-Distill-Qwen-32B. Expected Calibration Errors (ECE) remained under 0.1, showcasing high reliability. For instance, R1-Distill-Qwen-32B had an ECE of just 0.01 on GSM8K and 0.06 on MATH.

性能结果清晰可量化。对于使用R1-Distill-Qwen-32b之类的模型时，对于某些数据集（例如AIME），探针的ROC-AUC得分超过了0.9。预期校准误差（ECE）保持在0.1之下，显示了高可靠性。例如，R1-DISTILL-QWEN-32B的ECE在GSM8K上仅为0.01，数学上的ECE为0.06。

In application, the probe was used to implement a confidence-based early exit strategy during inference. The reasoning process was halted when the probe's confidence in an answer exceeded a threshold. At a confidence threshold of 0.85, the accuracy remained at 88.2%, while the inference token count was reduced by 24%. Even at a threshold of 0.9, accuracy stayed at 88.6%, with a 19% token reduction. Compared to static exit methods, this dynamic strategy achieved up to 5% higher accuracy using the same or fewer tokens.

在应用中，探测器在推断期间被用来实施基于置信的早期退出策略。当探测器对答案的信心超过阈值时，推理过程停止了。在0.85的置信度阈值下，准确性保持在88.2％，而推断令牌计数降低了24％。即使在0.9的阈值下，准确性仍保持在88.6％，降低了令牌的19％。与静态退出方法相比，使用相同或更少的令牌，这种动态策略的精度高达5％。

This study provides an efficient, integrated way for reasoning models to self-verify during inference. The researchers' approach highlights a gap—models inherently know when they're right, but they don't act on it. This research opens up avenues for smarter, more efficient reasoning systems by leveraging internal representations through probing. It demonstrates that tapping into what the model already "knows" can lead to significant improvements in both performance and resource use.

这项研究为推理模型在推理期间进行自我验证提供了一种有效的综合方法。研究人员的方法突出了一个差距 - 模型本质上知道它们何时正确，但他们不采取行动。这项研究通过探测利用内部表示，开辟了更智能，更有效的推理系统的途径。它表明，利用已经“知道”的模型可以导致性能和资源使用的重大改进。