Chain-of-Thought Reasoning

CoT in 2026: what still works, what got absorbed into reasoning models

Chain-of-Thought was introduced in Wei et al. 2022 ("Chain-of-Thought Prompting Elicits Reasoning in Large Language Models") and became the defining prompting technique of the first LLM wave. Four years later, the practitioner view on r/MachineLearning, r/LocalLLaMA, and r/ChatGPTPro is that CoT has bifurcated: partially obsolete for frontier reasoning models, still essential everywhere else.

What changed with reasoning models:

• →o-series, GPT-5 Thinking, Claude extended thinking, Gemini 2.5 Pro reasoning do CoT internally. You don't need to prompt "think step by step" — the model already does it, at scale, and hides the chain from the output. OpenAI's o1 announcement was explicit: the technique moved from prompt to architecture.
• →On reasoning models, explicit CoT prompts can hurt. Adding "think step by step" to a reasoning model sometimes anchors the internal chain to a worse path. Let the model do its thing.
• →On non-reasoning models, CoT is still the single biggest accuracy lever. For Claude Haiku, GPT-4o mini, Gemini Flash, Llama 3, and fine-tuned open models, explicit CoT still produces the 30-60 % gains the 2022 paper reported.

What still works as practitioners have learned:

• →"Let's think step by step" vs. structured CoT. The naive zero-shot trigger (from Kojima et al. 2022) helps, but structured CoT ("First identify what's asked; then list known facts; then compute; then verify") outperforms it consistently.
• →Self-consistency amplifies CoT on small models. Sample 5-10 chains at T=0.7, take the majority. The cost/quality tradeoff is best for cheap models on hard tasks.
• →CoT fails on tasks without verifiable answers. Majority voting and chain-checking assume a correct answer exists. For open-ended generation, CoT mostly produces longer outputs without quality gains.
• →Verifier-based selection beats voting. When possible, run code, check types, query a database. External verification is always better than LLM-majority.

Common 2026 mistakes:

• →Using CoT on reasoning models without testing. It can help, hurt, or do nothing. Benchmark on your actual task.
• →Hiding the chain from users who need auditability. For regulated domains, the chain is the audit trail. Don't discard it.
• →Mistaking longer output for better reasoning. More tokens aren't more thought. Chains that restate the prompt add latency without gain.

The honest framing: Chain-of-Thought is the most successful prompting technique in LLM history, and in 2026 its role has moved. For frontier reasoning models, it's architectural; for everything else, it's still the default way to extract correct multi-step reasoning. Don't assume one-size-fits-all; benchmark both approaches on your task before committing.

Why Models Need Help Reasoning

LLMs predict the next token, they do not "reason" in the human sense. For simple questions, direct prediction works fine. But for multi-step problems (math, logic, analysis), the model needs to lay out intermediate steps to arrive at the correct answer.

Think of it this way: if someone asks you "What is 47 times 83?", you do not instantly produce "3,901." You decompose: 47 times 80 = 3,760, plus 47 times 3 = 141, total = 3,901. Chain-of-Thought forces the model to decompose in the same way.

The Three CoT Techniques

Zero-Shot CoT: The Magic Words

A bookstore has 3 shelves. Each shelf has 4 sections. Each section holds 15 books. If the store is 70% full, how many books are in the store?

Few-Shot CoT: Teaching Reasoning by Example

Q: If a train travels 120km in 2 hours, and then 90km in 1.5 hours, what is the average speed for the whole trip?

A: Let me work through this step by step.
1. Total distance = 120 + 90 = 210 km
2. Total time = 2 + 1.5 = 3.5 hours
3. Average speed = 210 / 3.5 = 60 km/h
Answer: 60 km/h

Self-Consistency: Voting for the Best Answer

When CoT Fails

Test Your Understanding

Further Exploration

You have mastered Chain-of-Thought and Self-Consistency. In the next article, you will explore Tree-of-Thought, a technique that lets the model explore and backtrack through branching reasoning paths, solving problems that linear reasoning cannot.

Continue to Tree-of-Thought Reasoning Arena to go beyond linear thinking.

CoT vs ToT: The Key Difference

How Tree-of-Thought Works

Problem: Plan a 3-day marketing campaign.

Subproblems:
1. Define target audience and messaging
2. Choose channels and budget allocation
3. Create content calendar and metrics

The Reasoning Arena Pattern

A powerful way to implement ToT is through a Reasoning Arena: you prompt the AI to take on multiple roles and debate.

You will act as 3 experts debating the best solution:

Expert A: Conservative strategist (risk-averse, proven methods)
Expert B: Innovation advocate (cutting-edge, high risk-reward)
Expert C: Data analyst (evidence-first, quantitative)

When to Use Tree-of-Thought

Limitations and Practical Concerns

1. →Cost: ToT uses 5-20x more API calls than CoT. Budget accordingly.
2. →Latency: Multiple sequential calls mean longer wait times. Not suitable for real-time interactions.
3. →Complexity: Implementing ToT requires orchestration logic (which branch to expand, when to prune).
4. →Diminishing returns: For well-defined problems with clear steps, CoT is faster and equally accurate.
5. →Model dependency: Small models produce incoherent branches. ToT works best with frontier models.

Test Your Understanding

Continue Learning

You now command the full reasoning toolkit: direct prompting, Chain-of-Thought, Self-Consistency, and Tree-of-Thought. In the next module, you will learn to chain and route prompts, building multi-step pipelines that orchestrate these techniques together.

Continue to Prompt Chaining and Pipelines to build your first AI workflow.