How AI Agents Plan: From Chain-of-Thought to Tree Search

Ask an AI agent to "refactor this module, update the tests, and open a pull request," and something has to decide the order of operations, notice when a step fails, and figure out what to try instead. That something is the agent's planning mechanism — and it is usually the least visible, least understood part of the system. Most people interacting with agents see the final answer or the executed action. They rarely see the internal scaffolding that decided which action to take, in what order, and why.

Planning is not a single technique. It's a spectrum ranging from "generate one paragraph of reasoning, then answer" to "explore hundreds of candidate action sequences and pick the best one, with backtracking along the way." Understanding where a given agent sits on that spectrum tells you a lot about what it can reliably do, where it will fail silently, and how much compute a task is actually going to cost.

This guide walks through AI agent planning from the simplest approach to the most expensive: chain-of-thought, the ReAct loop, plan-and-execute, and search-based methods like Tree of Thoughts and Monte Carlo Tree Search. It compares them side by side, then covers what each means for builders choosing an architecture and where all of them still fall short.

What "planning" means for an LLM agent

A large language model, by itself, has no persistent goal and no built-in notion of a multi-step task. It receives a prompt and produces a continuation. Planning is the layer built on top of that raw capability — a loop or structure that repeatedly calls the model, feeds it the results of prior steps, and decides when to stop.

At minimum, an agent's planning loop needs to do four things:

1. Decompose a goal into smaller steps or sub-goals.
2. Select the next action given the current state.
3. Execute that action (call a tool, run code, query an API, write text).
4. Evaluate the result and decide whether to continue, retry, backtrack, or stop.

The differences between planning architectures come down to how much structure and search is applied to steps 1 and 2, and how rigorously step 4 is checked before moving on. A chatbot that answers a question in one shot skips this loop entirely. An agent that books a multi-leg trip, debugs a failing test suite, or coordinates a multi-file code change cannot.

The baseline: chain-of-thought

Chain-of-thought (CoT) prompting is the simplest form of planning, and it's the one most people have encountered even if they didn't know its name. Instead of asking a model to answer directly, you ask it to "think step by step" before producing a final answer. The model generates a sequence of intermediate reasoning tokens, and because each token is conditioned on the ones before it, this effectively lets the model build up a chain of sub-conclusions rather than jumping straight to a guess.

CoT is planning in the loosest sense: the model plans by narrating a linear sequence of thoughts, once, without revisiting earlier steps. There is no branching, no backtracking, and no external verification. If the model reasons its way into an error at step 3, that error propagates through every subsequent step, because the model has committed to that thread and rarely reconsiders it.

This works surprisingly well for problems where the correct reasoning path is fairly obvious once you start looking for it — arithmetic word problems, straightforward code explanations, simple multi-hop question answering. It works poorly for problems where the first plausible-looking approach is a dead end, because CoT has no mechanism for noticing that and trying something else.

From reasoning to acting: the ReAct pattern

Chain-of-thought only produces text. It doesn't touch the outside world. The next major step in agent planning was combining reasoning with tool use — letting the model interleave "thinking" tokens with actual actions (a search query, a calculator call, a file read) and then feed the results of those actions back into its context before continuing to reason.

This pattern, commonly called ReAct (Reason + Act), turns the model's internal monologue into an observable loop:

• Thought: the model states what it believes and what it needs to find out.
• Action: it calls a tool — search, code execution, an API — based on that thought.
• Observation: the tool's output is appended to the context — a process closely tied to context engineering, deciding what actually belongs in the model's limited context window at each step.
• Repeat: the model incorporates the observation into its next thought, and continues until it has enough information to answer or has completed the task.

This is the architecture behind most of what people currently call "AI agents": a model in a loop, given tools — typically implemented through the function- and tool-calling APIs documented by providers like OpenAI — repeatedly reasoning about what to do next based on what just happened. It's a meaningful upgrade over pure chain-of-thought because the model's plan is now grounded in real feedback rather than its own unverified assumptions. If a search returns nothing useful, the model sees that and can adjust — something a closed-loop CoT chain cannot do.

The limitation is that ReAct is still fundamentally linear and greedy. At each step, the model commits to one action based on what looks best right now. It does not compare that action against alternatives, and once it moves forward, it generally doesn't reconsider unless something goes visibly wrong. For tasks with a single reasonably clear path forward, this greediness is fine. For tasks where an early choice can quietly foreclose a better outcome later — the classic problem in planning and search — greedy, one-shot decision-making is a real weakness.

Adding structure: decomposition and explicit planning

A step up from ReAct's implicit, step-by-step planning is to make planning an explicit, separate phase. Instead of deciding one action at a time, the model first produces a structured plan — a list of sub-tasks — and then executes (or delegates) each sub-task, potentially replanning if a sub-task fails or reveals new information.

This "plan-and-execute" style separates two jobs that ReAct conflates: figuring out what needs to happen and figuring out how to do the next single thing. Separating them has practical benefits:

• The plan itself becomes inspectable and editable — a human (or another model) can review it before execution starts, which matters for anything with real-world consequences.
• Sub-tasks can be delegated to smaller, cheaper models or to specialized tools, since only the planning step requires the most capable reasoning.
• Failure handling is cleaner: if sub-task 3 fails, the system can revise the remaining plan rather than restarting the whole reasoning process from scratch.

The tradeoff is that upfront plans are only as good as the model's ability to anticipate the task before doing any of it. Plans made in advance of execution tend to be wrong in the details — a well-known problem in classical robotics and operations research long before LLMs existed — so most practical implementations pair an explicit plan with a replanning step: execute, observe, and revise the remaining plan if the observation doesn't match what was expected.

Search-based planning: Tree of Thoughts and beyond

Both CoT and ReAct generate a single sequence of reasoning and commit to it. Tree-based planning methods explicitly generate and compare multiple candidate continuations at each step, then use some evaluation criterion to decide which branch to keep pursuing — abandoning the rest.

Tree of Thoughts (ToT) is the clearest example. At each step, instead of asking the model for one next thought, you ask it for several candidate next thoughts. A separate evaluation step — sometimes the same model asked to critique its own options, sometimes a different scoring mechanism — rates each candidate. The search proceeds along the most promising branches, using strategies borrowed directly from classical search algorithms:

• Breadth-first search: expand every branch at each level, keep only the top-scoring few, and continue.
• Depth-first search with backtracking: follow one branch as deep as it looks promising, and backtrack to try an alternative the moment it stops looking promising.
• Monte Carlo Tree Search (MCTS): sample many possible continuations, use random rollouts or scoring to estimate the value of each, and progressively concentrate the search on the branches with the best estimated outcomes — the same core algorithm behind AlphaGo's move selection, a technique studied extensively in the reinforcement-learning literature on arxiv.org, adapted to a token or action space instead of a board.

The advantage is straightforward: for tasks with multiple plausible approaches, only some of which pan out, tree search lets the agent explore more than one and abandon the losers before committing significant effort to them. This matters enormously for tasks like mathematical proof search, complex code generation with several viable implementation strategies, or planning problems with real dead ends (a puzzle, a routing problem, a multi-step negotiation).

The cost is equally straightforward: exploring multiple branches multiplies the number of model calls, often by an order of magnitude or more compared to a single linear pass. Tree search trades compute for reliability — a direct expression of the test-time compute tradeoff, spending more inference-time computation in exchange for a better answer — and that trade is only worth making when the task actually has meaningfully different branches worth comparing. Throwing tree search at a problem with one obvious solution path just burns tokens for no benefit.

A comparison of planning approaches

Benefits of AI Agent Planning

Multi-step goals become tractable

A raw model call answers one prompt. A planning loop lets the same model work through a goal that needs ten tool calls, three file edits, and a test run, because each step is decided with the results of the previous ones in view. Decomposition turns "ship this feature" into a sequence of checkable sub-tasks, which is the difference between an assistant that drafts and an agent that finishes.

Actions are grounded in real feedback

ReAct-style loops feed tool output back into the model's context before the next decision. Instead of reasoning from assumptions about what a file contains or what an API returns, the agent reasons from what actually came back. That grounding cuts a whole class of confident-but-wrong answers, and when a search returns nothing or a command errors, the agent sees it and can change course rather than narrating success.

Plans can be reviewed before anything happens

Separating planning from execution produces an artefact a person or policy engine can read. For tasks with consequences, such as spending money, emailing customers, or touching production data, that checkpoint is valuable on its own: a reviewer can strike a risky step or reorder work before the agent acts. It also makes post-incident analysis much easier, because the intended plan and the actual trace can be compared.

Compute can be spent where it pays off

Planning architectures give builders a dial. Routine tasks can run through a cheap linear loop; tasks with real dead ends can get branching and evaluation. Plan-and-execute also lets the expensive model do the planning while cheaper models or deterministic tools handle individual sub-tasks. Used deliberately, that dial keeps cost proportional to difficulty instead of paying tree-search prices for everything.

Errors can be caught before they compound

Search-based methods and explicit evaluation steps let an agent notice a bad branch and abandon it early. On long-horizon work, where a subtle mistake at step two can quietly ruin step nine, that capacity to back out is what keeps reliability from decaying as tasks get longer. Even a light version helps: generating two candidate approaches and asking a critic to pick one catches many errors that a single greedy pass would carry all the way to the end.

AI Agent Planning Use Cases

Coding agents working across a repository

A request like "refactor this module and update the tests" has an obvious order of operations and an equally obvious way to fail: change the code, break an import, and never notice. Coding agents typically pair a ReAct loop with an explicit verifier, running the test suite after each edit. The test results become the observation that drives replanning, so the agent fixes what it broke before opening a pull request.

Research tasks across many sources

Gathering information from dozens of documents or web pages is a decomposition problem: break the question into sub-questions, search for each, and reconcile the findings. Plan-and-execute fits well here because sub-questions can be researched independently, often by cheaper models, and the planner can add follow-up questions when an answer reveals a gap. The output is a sourced synthesis rather than a single-pass guess.

Ticket triage and operational queues

Working through a queue of support or ops tickets means classifying each one, pulling context from internal systems, and choosing an action. The branching factor per ticket is usually low, so a grounded ReAct loop with good tools handles most of the work cheaply. An inspectable plan before any customer-facing action gives a human or policy check the chance to stop a bad reply.

Puzzles, proofs, and routing problems

Mathematical proof search, constraint puzzles, and routing problems have genuine dead ends: the first plausible approach often fails. These are the tasks where Tree of Thoughts and MCTS earn their cost, because there is a usable signal for scoring partial progress and the alternatives really are different. Here, comparing branches and backtracking is what produces a correct answer at all.

Multi-leg bookings and transactional workflows

Booking a multi-leg trip or coordinating a multi-step purchase involves dependencies (the hotel depends on the flight) and irreversible actions. Explicit planning surfaces the full sequence for approval before payment, and replanning handles the common case where a fare or room is no longer available at execution time. Because the plan names each booking step, a reviewer can see exactly what will be charged and in what order, and the agent can stop cleanly if an early leg falls through rather than leaving a stranded hotel reservation behind.

Why this matters right now

Interest in "AI agents" has outpaced a shared understanding of what's actually happening inside them. A large share of products marketed as agentic today are ReAct-style loops with a handful of tools — a real and useful architecture, but a specific one, with specific failure modes. Buyers and builders evaluating agent products benefit from being able to ask a concrete question: does this system plan once and act, or does it search, compare, and revise? The answer predicts a lot about reliability, cost, and latency before you've run a single test case.

This distinction has become more practically important as agents have moved from single-turn assistants toward systems expected to complete open-ended, multi-hour tasks with minimal supervision — writing a feature end to end, triaging a queue of tickets, running a research task across dozens of sources. The longer and more open-ended the task, the more a purely greedy, non-backtracking plan compounds small errors into large failures, because there's more room for an early bad decision to go unnoticed until much later. Systems built for long-horizon autonomy increasingly need at least some capacity to notice they're on a bad path and revise, which is exactly the capability that simple CoT and single-pass ReAct loops lack.

Common AI Agent Planning Mistakes

Reaching for tree search by default

Tree of Thoughts and MCTS sound like upgrades, so teams adopt them before measuring whether a linear loop was failing. On tasks with one obvious path, the extra branches just multiply model calls and latency. The right order is the reverse: start with a grounded ReAct loop, find where it gets stuck in the logs, and add search only there.

Shipping search without a trustworthy evaluator

A tree search is steered entirely by how it scores partial solutions. When the scorer is the same model asked "does this look good?", the search tends to favour fluent, plausible branches rather than correct ones. Without a real check, such as tests, schema validation, or a separate critic with clear criteria, more search simply finds more convincing wrong answers.

Treating every action as reversible

Planning methods borrowed from games assume you can step back to a clean state. Agents that send messages, write to databases, or run shell commands with side effects cannot. Letting a backtracking planner explore branches that include irreversible tool calls is a recipe for duplicate emails and half-applied changes. Irreversible actions need to be flagged and gated, not explored.

Trusting the upfront plan too much

Plans written before any execution are usually wrong in the details. Teams that execute a fixed plan without a replanning step end up with agents that faithfully complete steps that no longer make sense once step one revealed new information. Every plan-and-execute loop needs an observe-and-revise point after each sub-task.

Logging only the final answer

When an agent gets something wrong, the final output rarely explains why. Without the thought, action, and observation trace, you cannot tell whether the model reasoned badly or reasoned well from bad data, and those two problems need completely different fixes.

AI Agent Planning Best Practices

If you are building or evaluating an agentic system, the planning architecture is one of the first things worth pinning down, because it constrains what you can promise users and how you should budget compute.

• Match the planning method to the task's branching factor. If there is usually one clear next step, a ReAct loop is often sufficient and dramatically cheaper than tree search. If the task genuinely has multiple viable approaches with different success rates, invest in structured comparison — even something as simple as generating two candidate solutions and having the model pick the better one is a cheap approximation of tree search.
• Make failure detection explicit, not implicit. A plan-and-execute or tree-search system is only as good as its evaluation step. If nothing verifies whether an intermediate result actually moved the task forward, backtracking has nothing to trigger on. Give the agent a way to check its own work — running tests, validating output against a schema, or a separate critique pass — before assuming a search-based architecture will catch its own mistakes.
• Budget for the cost multiplier. Tree-based methods can cost several times more in model calls than a linear loop for the same task. That's a legitimate trade for tasks where reliability matters more than latency or cost, and a bad trade otherwise. Measure before committing to the more expensive architecture — many tasks that look like they need search actually resolve fine with a single well-grounded ReAct pass plus a retry-on-failure loop.
• Keep the plan inspectable. For anything with real consequences — spending money, sending communications, modifying production data — an explicit plan-and-execute structure that surfaces the plan before execution gives a human (or an automated policy check) a chance to intervene. A pure ReAct loop that reasons and acts in the same breath doesn't offer that checkpoint.
• Instrument the loop, not just the output. Log each thought, action, and observation. When an agent produces a wrong final answer, the planning trace is usually where you find out whether the model reasoned correctly and acted on bad information, or reasoned incorrectly from the start — and those two failures need different fixes.

Limitations and open questions

None of the planning methods described above solve the underlying problem cleanly, and it's worth being honest about where they fall short.

Evaluation is the hard part, not search. Tree of Thoughts and MCTS both depend on being able to score a partial solution's quality before it's finished — but for many real tasks (is this half-written function going to work? is this negotiation strategy going to land?) a good mid-process evaluator is at least as hard to build as the task itself, an issue closely related to the reward-verification questions covered in how reasoning models are trained with verifiable rewards. Search only helps to the extent that the evaluation function is trustworthy; a tree search guided by a bad scorer just explores more paths to the wrong answer, more expensively.

Backtracking has a state problem. In classical search, backtracking means returning to a clean prior state. In agentic systems that call real tools — sending an email, writing to a database, running a shell command with side effects — many actions cannot be cleanly undone. Planning architectures borrowed from board games and pure text generation don't automatically account for the fact that some steps are irreversible, and treating them as reversible is a real source of production incidents.

More search does not always mean more correctness. Past a certain point, adding branches and depth to a search runs into the same diminishing returns seen in classical planning: exploring more of a search space that's poorly specified just finds more plausible-looking wrong answers. The quality ceiling is often set by the model's underlying competence at the task, not by how much search is layered on top.

There's no settled standard for how planning traces should be represented or shared between agents. As multi-agent systems become more common — one agent's plan feeding into another's execution — the lack of a common format for describing "here is my plan, here is my confidence in each step, here is what I've already ruled out" makes coordination between agents harder than it needs to be. This is an active area of protocol design rather than a solved problem.

What to watch next

A few threads are worth tracking if you want to stay ahead of how agent planning evolves:

1. Verifier and critic models built specifically for evaluation, separate from the model doing the planning — since evaluation quality, not search breadth, is usually the bottleneck in tree-based methods, and it's the same bottleneck we cover in how AI agent evaluation works.
2. Hybrid architectures that default to cheap, linear ReAct-style loops and escalate to structured search only when the agent detects it's stuck — getting most of the reliability benefit of tree search without paying its cost on every task.
3. Standardized formats for exposing an agent's plan and reasoning trace to other agents, tools, and human reviewers, which would make multi-agent coordination and human oversight considerably easier than today's ad hoc logging.
4. Better handling of irreversible actions inside planning loops — treating "can this be undone" as a first-class property of an action, not an afterthought handled by whoever wrote the tool.

FAQ

What's the difference between chain-of-thought and an AI agent?

Chain-of-thought is a prompting technique that gets a model to reason step by step before answering — it produces text, not action. An AI agent uses a planning loop (which may or may not include chain-of-thought) to decide on and execute actions, like calling tools or APIs, based on that reasoning. CoT can be one ingredient inside an agent's planning process, but by itself it isn't an agent.

Is ReAct the same thing as tree of thoughts?

No. ReAct interleaves reasoning with tool calls in a single, mostly linear sequence — it commits to one action at a time based on the most recent observation. Tree of Thoughts explicitly generates multiple candidate next steps, evaluates them, and searches across branches, abandoning ones that look unpromising. ReAct is cheaper and works well for grounded, mostly-linear tasks; tree search is more expensive but handles tasks with genuine dead ends better.

Why do AI agents fail on multi-step tasks even when each step seems reasonable?

Most production agents use greedy, linear planning (ReAct-style), which commits to each action based only on the current state without comparing it to alternatives or verifying it against the eventual goal. If an early step looks reasonable but is subtly wrong, nothing in a purely linear loop catches it until a much later step fails outright — by which point the error has already compounded.

Does adding tree search or MCTS always make an agent more reliable?

No. Search only helps when there's a trustworthy way to evaluate partial progress and when the task actually has multiple meaningfully different paths worth comparing. If the evaluation function is unreliable, search just explores more wrong answers at higher cost. For tasks with one clear approach, search adds expense without adding accuracy.

What is Monte Carlo Tree Search doing in an LLM agent?

MCTS samples many possible action sequences, uses rollouts or a scoring model to estimate how promising each one is, and progressively focuses further search on the best-performing branches. It's the same core algorithm used in game-playing systems like AlphaGo, adapted so that "moves" are actions or reasoning steps rather than board positions. It's expensive but useful for deep search spaces where a clear win/loss or quality signal exists.

How much more expensive is tree-based planning compared to a simple agent loop?

It varies by implementation, but generating and evaluating multiple candidates at each step commonly means several times more model calls than a single linear pass — sometimes an order of magnitude more for deep searches. That's why most production systems default to a cheaper linear loop and reserve search-based planning for tasks where the extra reliability is worth the extra compute.

Can an agent recover from a mistake without an explicit backtracking mechanism?

Only partially. A ReAct-style loop can adjust once it observes a clearly failed action (a tool error, an unexpected result), because that failure becomes part of its next input. But it has no way to recognize that an earlier, superficially successful step was actually the wrong choice — that requires either explicit backtracking, a verification step, or a human catching it.

Conclusion

Every agent has a planning mechanism, even when nobody designed one on purpose. A model in a loop with tools is already making decisions about order, retries, and when to stop, and the quality of those decisions sets the ceiling on what the agent can be trusted with. The useful mental model is a spectrum: chain-of-thought plans once and never looks back, ReAct grounds each step in real feedback but stays greedy, plan-and-execute makes the plan reviewable, and tree search compares alternatives at a real cost in model calls.

Two caveats matter more than the choice of technique. First, search is only as good as the evaluator guiding it, and for many real tasks a reliable mid-process evaluator is the hardest thing to build. Second, backtracking assumes you can undo things, and agents that send emails or write to databases often cannot. Design for irreversible actions explicitly rather than borrowing assumptions from board games.

Start with the cheapest architecture that handles your task, instrument every thought, action, and observation, and escalate to structured search only where the logs show a linear loop getting stuck. If you're deciding which planning approach fits a specific product or workflow, our AI agent development services or the team at Woyce Technologies can help you scope and build it.