# Beating Reasoning Models at 30% of the Cost: How Token-Level Entropy Enables Smarter AI Loops ## The $15/Million Token Problem Reasoning models have shown impressive capabilities, but they come with a higher price - often 2.75-3x more expensive than standard models. At $15 per million tokens versus $0.40-2.50 for standard models, this cost difference matters at scale. But what if we could achieve similar quality without the premium price? That's exactly what we've built. **[Try the interactive notebook →](https://colab.research.google.com/github/monostate/weave-logprobs-reasoning-loop/blob/main/wb-logprobs.ipynb)** **[View the code on GitHub →](https://github.com/monostate/weave-logprobs-reasoning-loop)** *[Read our full paper on arXiv →](soon)* *(pending review)* ## The Discovery: Models Know When They Don't Know Here's the key insight: when language models generate text, they actually produce probability distributions for each token. Most of the time, this rich information gets thrown away - we just take the most likely token and move on. But hidden in those probabilities is something valuable: **uncertainty signals**. When a model is confident, it might assign 95% probability to one token. When it's uncertain, that probability might spread across multiple options - maybe 30% for one word, 25% for another, 20% for a third. This spreading of probability is measurable through a concept called **entropy**. ## From Theory to Practice: The Entropy-Guided Loop Instead of always using expensive reasoning models, our system works like this: ```mermaid graph TB A[Input Query] --> B[Generate with Logprobs] B --> C{Calculate Metrics} C --> D[Perplexity > 1.4?] C --> E[Max Entropy > 1.5?] C --> F[Low Conf Tokens ≥ 3?] D -->|Yes| G[Trigger Loop] E -->|Yes| G F -->|Yes| G D -->|No| H[Output Original] E -->|No| H F -->|No| H G --> I[Build Uncertainty Report] I --> J[Show Uncertain Tokens
+ Alternatives] J --> K[Loop Pass] K --> L[Output Improved] style C fill:#fff,stroke:#000,color:#000 style G fill:#fff,stroke:#000,color:#000 style I fill:#fff,stroke:#000,color:#000 ``` ### 1. Generate and Measure The model generates an answer while we track the entropy (uncertainty) of each token: ```python # High entropy example - model is uncertain: "The capital of Australia is [Sydney: 45%, Melbourne: 30%, Canberra: 25%]" # Low entropy example - model is confident: "The capital of France is [Paris: 98%, Lyon: 1%, Marseille: 1%]" ``` ### 2. Smart Triggering We use multiple signals to decide if the loop is needed: - **Overall perplexity** > 1.4 (general uncertainty) - **Maximum token entropy** > 1.5 (critical uncertainty point) - **Low-confidence tokens** ≥ 3 (distributed uncertainty) This multi-metric approach catches different types of errors that single metrics miss. ### 3. Detailed Feedback Here's where we innovate: instead of just asking the model to "try again," we show it exactly where it was uncertain. ```mermaid graph LR A[Token Generation] --> B{Entropy Check} B -->|High| C["likely"
32.1% confidence] B -->|High| D["2030"
41.2% confidence] B -->|Low| E["is"
98% confidence] C --> F[Alternatives:
unlikely 28.5%
possible 22.3%] D --> G[Alternatives:
2040 31.5%
2035 15.8%] E --> H[No alternatives
needed] F --> I[Include in
Uncertainty Report] G --> I I --> J[Targeted
Loop] style B fill:#fff,stroke:#000,color:#000 style C fill:#fff,stroke:#000,color:#000 style D fill:#fff,stroke:#000,color:#000 style E fill:#fff,stroke:#000,color:#000 ``` The model receives: ``` === UNCERTAINTY ANALYSIS === Most uncertain tokens: 1. Token: 'likely' (position 15) Context: ...is [likely] to... Confidence: 32.1%, Entropy: 1.43 Alternatives: 'unlikely' (28.5%), 'possible' (22.3%) ``` The model sees its own uncertainty and can make informed corrections. ## Real Results: Cost, Quality, and Speed Testing on controversial and complex questions shows: | Model | Cost per Query | Quality | When to Use | |-------|---------------|---------|-------------| | GPT-4.1-mini + EGL | $0.0007-0.0011 | 95% match | Cost-sensitive applications | | o4-mini (reasoning) | $0.0019-0.0058 | Baseline | Maximum accuracy needed | | **Cost Efficiency** | **30-40%** | **Comparable** | **Most use cases** | Example from our tests: - **Question**: "Is artificial general intelligence likely to be achieved by 2030?" - **Our approach**: Triggered loop due to high entropy on "likely" and "2030" - **Result**: Nuanced answer acknowledging uncertainty, similar quality to o4-mini - **Savings**: 70% cost reduction ## Why This Matters: The Signal-Action Mismatch Our research uncovered why nobody had done this before - what we call the "signal-action mismatch": ```mermaid graph TD A[Traditional Approach] --> B[Calculate Entropy] B --> C{High Entropy?} C -->|Yes| D[Know Something Wrong] C -->|No| E[Continue] D --> F[But What Exactly?] F --> G[Regenerate Entire Response] G --> H[Expensive & Unfocused] I[Our Approach: EGL] --> J[Calculate Entropy] J --> K{High Entropy?} K -->|Yes| L[Identify Exact Tokens] K -->|No| M[Continue] L --> N[Extract Alternatives] N --> O[Build Uncertainty Report] O --> P[Targeted Loop] P --> Q[Efficient & Precise] style H fill:#fff,stroke:#000,color:#000 style Q fill:#fff,stroke:#000,color:#000 ``` **The Problem**: - Token entropy is cheap to calculate but noisy - Loop passes are expensive (requires full model passes) - Previous work either used expensive signals (semantic judges) or avoided loops entirely **Our Solution**: Make the cheap signal actionable by providing detailed, interpretable feedback. Instead of just knowing "something is wrong," the model knows exactly which tokens are uncertain and what the alternatives were. ## The Research Gap We're Filling Our approach builds on recent debates about reasoning effectiveness. While Song et al. (2025) show that tool augmentation makes reasoning models effective, we demonstrate an orthogonal approach through uncertainty awareness. After extensive analysis, we confirmed our approach is novel: 1. **CURE (2024)** - Closest prior work, but only passes prefixes without alternatives 2. **CALM (2022)** - Uses entropy for the OPPOSITE (reducing computation when confident) 3. **Self-Consistency** - Generates many paths and votes, doesn't refine a single path 4. **Self-Refine** - Uses qualitative feedback, no entropy metrics **We're the first to combine**: - Token-level entropy as a loop trigger - Detailed uncertainty feedback with alternatives - Multi-metric OR logic for triggering - Direct comparison with reasoning models ## Try It Yourself We've open-sourced everything with [Weights & Biases Weave](https://wandb.ai/site/weave) integration for full observability: ```python # Install and run pip install -r requirements.txt python wb-logprobs.py # Or try the notebook # https://colab.research.google.com/github/monostate/weave-logprobs-reasoning-loop/blob/main/wb-logprobs.ipynb ``` The system shows you exactly how it makes decisions: - Which tokens triggered the loop - What alternatives were considered - How confidence changed after the loop - Full cost and performance metrics ## What's Next: Benchmarks and Beyond Our paper presents results on standard benchmarks and technical queries: - **ARC-AGI** - Abstract reasoning - **LogiQA 2.0** - Logical reasoning - **GSM8K** - Math problems Results show we achieve 94.7% quality relative to reasoning models at 30-40% of the cost, with the loop triggered in only 31% of cases. Our paper includes: - Full benchmark results - Theoretical analysis of why this works - Ablation studies on each component - Comparison with 25+ related approaches ## The Bigger Picture: Democratizing AI Quality Expensive reasoning models create a divide - those who can afford premium AI and those who can't. By achieving similar quality at 30% of the cost, we're making high-quality AI generation accessible to more applications and users. This isn't just about saving money. It's about enabling new use cases where the previous cost was prohibitive: - Educational tools that can afford to be more helpful - Open-source projects with limited budgets - Researchers exploring new applications - Startups competing with larger players ## Technical Deep Dive For those interested in the implementation details: ```python # Core implementation snippet def calculate_entropy(top_k_alternatives): """Calculate Shannon entropy from token alternatives""" probs = [math.exp(logprob) for token, logprob in top_k_alternatives] total = sum(probs) probs = [p/total for p in probs] entropy = 0.0 for p in probs: if p > 0: entropy -= p * math.log(p) return entropy def should_loop(metrics): """Multi-metric OR logic for loop trigger""" return ( metrics['perplexity'] > 1.4 or metrics['max_entropy'] > 1.5 or metrics['low_conf_tokens'] >= 3 ) ``` ### The Core Metrics **Perplexity**: `exp(-mean(log_probabilities))` - Overall uncertainty measure **Shannon Entropy**: `-sum(p * log(p))` - Information-theoretic uncertainty per token **Confidence Distribution**: Percentage of tokens below confidence thresholds ### The Loop Prompt We don't just say "fix this." We provide: 1. The original question and draft answer 2. Detailed uncertainty analysis with specific tokens 3. Alternative tokens the model considered 4. Context around uncertain points 5. Explicit instructions for resolution ### Why It Works - **Uncertainty identifies struggle points**: High entropy correlates with potential errors - **Alternatives guide correction**: Seeing other options helps reconsideration - **Context prevents over-correction**: Surrounding text maintains coherence - **Multi-metric catches different failures**: Perplexity for overall issues, entropy for specific points ## Join the Research This work is part of a larger effort to make AI more reliable through uncertainty awareness. We're looking for: - Researchers to collaborate on benchmarks - Engineers to integrate into production systems - Feedback on real-world applications **Get involved**: - [GitHub Repository](https://github.com/monostate/weave-logprobs-reasoning-loop) - [Interactive Notebook](https://colab.research.google.com/github/monostate/weave-logprobs-reasoning-loop/blob/main/wb-logprobs.ipynb) - Email: andrew@monostate.ai ## Conclusion: The Future of AI Loops We've shown that models can fix their own mistakes when given the right information about their uncertainty. This isn't just a cost optimization - it's a fundamental insight about how to make AI systems more reliable. By treating uncertainty as actionable information rather than noise, we're opening new possibilities for self-improving AI systems that are both capable and affordable. The days of choosing between cost and quality may be coming to an end. With entropy-guided loops, we can have both. --- *Andrew Correa and Ana Carolina H. Matos are researchers at Monostate, focusing on making AI more reliable through uncertainty quantification. This research was conducted with support from the Weights & Biases Weave team.* *Our paper "Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation" is available on arXiv (pending). For collaboration inquiries, contact andrew@monostate.ai or carol@monostate.ai*