Exploring the Hidden States of Chain-of-Thought Reasoning Models to Reduce Inference Inefficiency

Artificial intelligence systems have made significant strides in simulating human-style reasoning, particularly mathematics and logic. These models don't just generate answers—they walk through a series of logical steps to reach conclusions, offering insights into how and why those answers are produced. This step-by-step reasoning, often called Chain-of-Thought (CoT), has become vital in how machines handle complex problem-solving tasks.

Artificial intelligence systems have made remarkable progress in simulating human-style reasoning, especially in domains like mathematics and logic. Unlike typical generative models, these systems generate a series of intermediate steps to reach a final answer, offering insights into the reasoning process. This step-by-step reasoning, often called Chain-of-Thought (CoT), is crucial for machines to handle complex problem-solving tasks.

A common challenge researchers face is the models' inefficiency during inference. The reasoning models may continue processing even after attaining a correct conclusion, leading to overthinking. This generates unnecessary tokens, increasing computational cost.

Many current approaches measure a model's confidence using verbal prompts or by analyzing multiple outputs. These "black-box" strategies ask the model to report how sure it is of its answer. However, they are often imprecise and computationally expensive. On the other hand, "white-box" methods investigate models' internal hidden states to extract signals that may correlate with answer correctness.

Prior work has shown that a model's internal states can indeed indicate the validity of final answers. However, applying this to intermediate steps in long reasoning chains is still an underexplored direction.

To bridge this gap, a team of researchers from New York University and NYU Shanghai designed a lightweight probe—a simple two-layer neural network—to inspect a model's hidden states at intermediate reasoning steps. Their models of choice were the DeepSeek-R1-Distill series and QwQ-32B, known for their excellent step-by-step reasoning capabilities, tested across various datasets including AIME, GSM8K, and MATH. The researchers trained their probe to read the internal state associated with each chunk of reasoning and predict whether the current intermediate answer was correct.

To construct their approach, they segmented each long CoT output into smaller parts or chunks, using markers like "wait" or "verify" to identify breaks in reasoning. They used the last token's hidden state in each chunk as a representation and matched this to a correctness label, which was judged using another model. These representations were then used to train the probe on binary classification tasks. The probe was fine-tuned using grid search across hyperparameters like learning rate and hidden layer size, with most models converging to linear probes—highlighting that correctness information is often linearly embedded in the hidden states.

The probe worked effectively for fully formed answers and even showed the ability to predict correctness before an answer was completed, alluding to look-ahead capabilities.

Performance results were clear and quantifiable. The probes achieved ROC-AUC scores exceeding 0.9 for some datasets like AIME when using models like R1-Distill-Qwen-32B. Expected Calibration Errors (ECE) remained under 0.1, showcasing high reliability. For instance, R1-Distill-Qwen-32B had an ECE of just 0.01 on GSM8K and 0.06 on MATH.

In application, the probe was used to implement a confidence-based early exit strategy during inference. The reasoning process was halted when the probe's confidence in an answer exceeded a threshold. At a confidence threshold of 0.85, the accuracy remained at 88.2%, while the inference token count was reduced by 24%. Even at a threshold of 0.9, accuracy stayed at 88.6%, with a 19% token reduction. Compared to static exit methods, this dynamic strategy achieved up to 5% higher accuracy using the same or fewer tokens.

This study provides an efficient, integrated way for reasoning models to self-verify during inference. The researchers' approach highlights a gap—models inherently know when they're right, but they don't act on it. This research opens up avenues for smarter, more efficient reasoning systems by leveraging internal representations through probing. It demonstrates that tapping into what the model already "knows" can lead to significant improvements in both performance and resource use.