Ask an AI agent to "refactor this module, update the tests, and open a pull request," and something has to decide the order of operations, notice when a step fails, and figure out what to try instead. That something is the agent's planning mechanism — and it is usually the least visible, least understood part of the system. Most people interacting with agents see the final answer or the executed action. They rarely see the internal scaffolding that decided which action to take, in what order, and why.
Planning is not a single technique. It's a spectrum ranging from "generate one paragraph of reasoning, then answer" to "explore hundreds of candidate action sequences and pick the best one, with backtracking along the way." Understanding where a given agent sits on that spectrum tells you a lot about what it can reliably do, where it will fail silently, and how much compute a task is actually going to cost.
What "planning" means for an LLM agent
A large language model, by itself, has no persistent goal and no built-in notion of a multi-step task. It receives a prompt and produces a continuation. Planning is the layer built on top of that raw capability — a loop or structure that repeatedly calls the model, feeds it the results of prior steps, and decides when to stop.
At minimum, an agent's planning loop needs to do four things:
- Decompose a goal into smaller steps or sub-goals.
- Select the next action given the current state.
- Execute that action (call a tool, run code, query an API, write text).
- Evaluate the result and decide whether to continue, retry, backtrack, or stop.
The differences between planning architectures come down to how much structure and search is applied to steps 1 and 2, and how rigorously step 4 is checked before moving on. A chatbot that answers a question in one shot skips this loop entirely. An agent that books a multi-leg trip, debugs a failing test suite, or coordinates a multi-file code change cannot.
The baseline: chain-of-thought
Chain-of-thought (CoT) prompting is the simplest form of planning, and it's the one most people have encountered even if they didn't know its name. Instead of asking a model to answer directly, you ask it to "think step by step" before producing a final answer. The model generates a sequence of intermediate reasoning tokens, and because each token is conditioned on the ones before it, this effectively lets the model build up a chain of sub-conclusions rather than jumping straight to a guess.
CoT is planning in the loosest sense: the model plans by narrating a linear sequence of thoughts, once, without revisiting earlier steps. There is no branching, no backtracking, and no external verification. If the model reasons its way into an error at step 3, that error propagates through every subsequent step, because the model has committed to that thread and rarely reconsiders it.
This works surprisingly well for problems where the correct reasoning path is fairly obvious once you start looking for it — arithmetic word problems, straightforward code explanations, simple multi-hop question answering. It works poorly for problems where the first plausible-looking approach is a dead end, because CoT has no mechanism for noticing that and trying something else.
From reasoning to acting: the ReAct pattern
Chain-of-thought only produces text. It doesn't touch the outside world. The next major step in agent planning was combining reasoning with tool use — letting the model interleave "thinking" tokens with actual actions (a search query, a calculator call, a file read) and then feed the results of those actions back into its context before continuing to reason.
This pattern, commonly called ReAct (Reason + Act), turns the model's internal monologue into an observable loop:
- Thought: the model states what it believes and what it needs to find out.
- Action: it calls a tool — search, code execution, an API — based on that thought.
- Observation: the tool's output is appended to the context.
- Repeat: the model incorporates the observation into its next thought, and continues until it has enough information to answer or has completed the task.
This is the architecture behind most of what people currently call "AI agents": a model in a loop, given tools, repeatedly reasoning about what to do next based on what just happened. It's a meaningful upgrade over pure chain-of-thought because the model's plan is now grounded in real feedback rather than its own unverified assumptions. If a search returns nothing useful, the model sees that and can adjust — something a closed-loop CoT chain cannot do.
The limitation is that ReAct is still fundamentally linear and greedy. At each step, the model commits to one action based on what looks best right now. It does not compare that action against alternatives, and once it moves forward, it generally doesn't reconsider unless something goes visibly wrong. For tasks with a single reasonably clear path forward, this greediness is fine. For tasks where an early choice can quietly foreclose a better outcome later — the classic problem in planning and search — greedy, one-shot decision-making is a real weakness.
Adding structure: decomposition and explicit planning
A step up from ReAct's implicit, step-by-step planning is to make planning an explicit, separate phase. Instead of deciding one action at a time, the model first produces a structured plan — a list of sub-tasks — and then executes (or delegates) each sub-task, potentially replanning if a sub-task fails or reveals new information.
This "plan-and-execute" style separates two jobs that ReAct conflates: figuring out what needs to happen and figuring out how to do the next single thing. Separating them has practical benefits:
- The plan itself becomes inspectable and editable — a human (or another model) can review it before execution starts, which matters for anything with real-world consequences.
- Sub-tasks can be delegated to smaller, cheaper models or to specialized tools, since only the planning step requires the most capable reasoning.
- Failure handling is cleaner: if sub-task 3 fails, the system can revise the remaining plan rather than restarting the whole reasoning process from scratch.
The tradeoff is that upfront plans are only as good as the model's ability to anticipate the task before doing any of it. Plans made in advance of execution tend to be wrong in the details — a well-known problem in classical robotics and operations research long before LLMs existed — so most practical implementations pair an explicit plan with a replanning step: execute, observe, and revise the remaining plan if the observation doesn't match what was expected.
Search-based planning: Tree of Thoughts and beyond
Both CoT and ReAct generate a single sequence of reasoning and commit to it. Tree-based planning methods explicitly generate and compare multiple candidate continuations at each step, then use some evaluation criterion to decide which branch to keep pursuing — abandoning the rest.
Tree of Thoughts (ToT) is the clearest example. At each step, instead of asking the model for one next thought, you ask it for several candidate next thoughts. A separate evaluation step — sometimes the same model asked to critique its own options, sometimes a different scoring mechanism — rates each candidate. The search proceeds along the most promising branches, using strategies borrowed directly from classical search algorithms:
- Breadth-first search: expand every branch at each level, keep only the top-scoring few, and continue.
- Depth-first search with backtracking: follow one branch as deep as it looks promising, and backtrack to try an alternative the moment it stops looking promising.
- Monte Carlo Tree Search (MCTS): sample many possible continuations, use random rollouts or scoring to estimate the value of each, and progressively concentrate the search on the branches with the best estimated outcomes — the same core algorithm behind AlphaGo's move selection, adapted to a token or action space instead of a board.
The advantage is straightforward: for tasks with multiple plausible approaches, only some of which pan out, tree search lets the agent explore more than one and abandon the losers before committing significant effort to them. This matters enormously for tasks like mathematical proof search, complex code generation with several viable implementation strategies, or planning problems with real dead ends (a puzzle, a routing problem, a multi-step negotiation).
The cost is equally straightforward: exploring multiple branches multiplies the number of model calls, often by an order of magnitude or more compared to a single linear pass. Tree search trades compute for reliability, and that trade is only worth making when the task actually has meaningfully different branches worth comparing — throwing tree search at a problem with one obvious solution path just burns tokens for no benefit.
A comparison of planning approaches
| Approach | Branching | Backtracking | Relative cost | Best suited for |
|---|---|---|---|---|
| Chain-of-thought | None (single path) | No | Low | Simple, mostly-linear reasoning tasks |
| ReAct (reason + act) | None, but grounded in tool feedback | Implicit, only on visible failure | Low–moderate | Tasks needing real-world information mid-reasoning |
| Plan-and-execute | Plan is fixed upfront, then linear execution | Replanning after failed sub-tasks | Moderate | Multi-step tasks with delegable sub-tasks |
| Tree of Thoughts | Multiple candidates per step | Explicit, evaluation-driven | High | Problems with several plausible approaches, some of which fail |
| Monte Carlo Tree Search | Sampled rollouts across many branches | Statistically guided | Very high | Deep search spaces with clear win/loss or scoring signals |
| Multi-agent debate/critique | Parallel independent attempts, then reconciliation | Cross-agent correction | High | Tasks where errors are easier to spot than to avoid |
Why this matters right now
Interest in "AI agents" has outpaced a shared understanding of what's actually happening inside them. A large share of products marketed as agentic today are ReAct-style loops with a handful of tools — a real and useful architecture, but a specific one, with specific failure modes. Buyers and builders evaluating agent products benefit from being able to ask a concrete question: does this system plan once and act, or does it search, compare, and revise? The answer predicts a lot about reliability, cost, and latency before you've run a single test case.
This distinction has become more practically important as agents have moved from single-turn assistants toward systems expected to complete open-ended, multi-hour tasks with minimal supervision — writing a feature end to end, triaging a queue of tickets, running a research task across dozens of sources. The longer and more open-ended the task, the more a purely greedy, non-backtracking plan compounds small errors into large failures, because there's more room for an early bad decision to go unnoticed until much later. Systems built for long-horizon autonomy increasingly need at least some capacity to notice they're on a bad path and revise, which is exactly the capability that simple CoT and single-pass ReAct loops lack.
Practical implications for builders
If you are building or evaluating an agentic system, the planning architecture is one of the first things worth pinning down, because it constrains what you can promise users and how you should budget compute.
- Match the planning method to the task's branching factor. If there is usually one clear next step, a ReAct loop is often sufficient and dramatically cheaper than tree search. If the task genuinely has multiple viable approaches with different success rates, invest in structured comparison — even something as simple as generating two candidate solutions and having the model pick the better one is a cheap approximation of tree search.
- Make failure detection explicit, not implicit. A plan-and-execute or tree-search system is only as good as its evaluation step. If nothing verifies whether an intermediate result actually moved the task forward, backtracking has nothing to trigger on. Give the agent a way to check its own work — running tests, validating output against a schema, or a separate critique pass — before assuming a search-based architecture will catch its own mistakes.
- Budget for the cost multiplier. Tree-based methods can cost several times more in model calls than a linear loop for the same task. That's a legitimate trade for tasks where reliability matters more than latency or cost, and a bad trade otherwise. Measure before committing to the more expensive architecture — many tasks that look like they need search actually resolve fine with a single well-grounded ReAct pass plus a retry-on-failure loop.
- Keep the plan inspectable. For anything with real consequences — spending money, sending communications, modifying production data — an explicit plan-and-execute structure that surfaces the plan before execution gives a human (or an automated policy check) a chance to intervene. A pure ReAct loop that reasons and acts in the same breath doesn't offer that checkpoint.
- Instrument the loop, not just the output. Log each thought, action, and observation. When an agent produces a wrong final answer, the planning trace is usually where you find out whether the model reasoned correctly and acted on bad information, or reasoned incorrectly from the start — and those two failures need different fixes.
Limitations and open questions
None of the planning methods described above solve the underlying problem cleanly, and it's worth being honest about where they fall short.
Evaluation is the hard part, not search. Tree of Thoughts and MCTS both depend on being able to score a partial solution's quality before it's finished — but for many real tasks (is this half-written function going to work? is this negotiation strategy going to land?) a good mid-process evaluator is at least as hard to build as the task itself. Search only helps to the extent that the evaluation function is trustworthy; a tree search guided by a bad scorer just explores more paths to the wrong answer, more expensively.
Backtracking has a state problem. In classical search, backtracking means returning to a clean prior state. In agentic systems that call real tools — sending an email, writing to a database, running a shell command with side effects — many actions cannot be cleanly undone. Planning architectures borrowed from board games and pure text generation don't automatically account for the fact that some steps are irreversible, and treating them as reversible is a real source of production incidents.
More search does not always mean more correctness. Past a certain point, adding branches and depth to a search runs into the same diminishing returns seen in classical planning: exploring more of a search space that's poorly specified just finds more plausible-looking wrong answers. The quality ceiling is often set by the model's underlying competence at the task, not by how much search is layered on top.
There's no settled standard for how planning traces should be represented or shared between agents. As multi-agent systems become more common — one agent's plan feeding into another's execution — the lack of a common format for describing "here is my plan, here is my confidence in each step, here is what I've already ruled out" makes coordination between agents harder than it needs to be. This is an active area of protocol design rather than a solved problem.
What to watch next
A few threads are worth tracking if you want to stay ahead of how agent planning evolves:
- Verifier and critic models built specifically for evaluation, separate from the model doing the planning — since evaluation quality, not search breadth, is usually the bottleneck in tree-based methods.
- Hybrid architectures that default to cheap, linear ReAct-style loops and escalate to structured search only when the agent detects it's stuck — getting most of the reliability benefit of tree search without paying its cost on every task.
- Standardized formats for exposing an agent's plan and reasoning trace to other agents, tools, and human reviewers, which would make multi-agent coordination and human oversight considerably easier than today's ad hoc logging.
- Better handling of irreversible actions inside planning loops — treating "can this be undone" as a first-class property of an action, not an afterthought handled by whoever wrote the tool.
FAQ
What's the difference between chain-of-thought and an AI agent?
Chain-of-thought is a prompting technique that gets a model to reason step by step before answering — it produces text, not action. An AI agent uses a planning loop (which may or may not include chain-of-thought) to decide on and execute actions, like calling tools or APIs, based on that reasoning. CoT can be one ingredient inside an agent's planning process, but by itself it isn't an agent.
Is ReAct the same thing as tree of thoughts?
No. ReAct interleaves reasoning with tool calls in a single, mostly linear sequence — it commits to one action at a time based on the most recent observation. Tree of Thoughts explicitly generates multiple candidate next steps, evaluates them, and searches across branches, abandoning ones that look unpromising. ReAct is cheaper and works well for grounded, mostly-linear tasks; tree search is more expensive but handles tasks with genuine dead ends better.
Why do AI agents fail on multi-step tasks even when each step seems reasonable?
Most production agents use greedy, linear planning (ReAct-style), which commits to each action based only on the current state without comparing it to alternatives or verifying it against the eventual goal. If an early step looks reasonable but is subtly wrong, nothing in a purely linear loop catches it until a much later step fails outright — by which point the error has already compounded.
Does adding tree search or MCTS always make an agent more reliable?
No. Search only helps when there's a trustworthy way to evaluate partial progress and when the task actually has multiple meaningfully different paths worth comparing. If the evaluation function is unreliable, search just explores more wrong answers at higher cost. For tasks with one clear approach, search adds expense without adding accuracy.
What is Monte Carlo Tree Search doing in an LLM agent?
MCTS samples many possible action sequences, uses rollouts or a scoring model to estimate how promising each one is, and progressively focuses further search on the best-performing branches. It's the same core algorithm used in game-playing systems like AlphaGo, adapted so that "moves" are actions or reasoning steps rather than board positions. It's expensive but useful for deep search spaces where a clear win/loss or quality signal exists.
How much more expensive is tree-based planning compared to a simple agent loop?
It varies by implementation, but generating and evaluating multiple candidates at each step commonly means several times more model calls than a single linear pass — sometimes an order of magnitude more for deep searches. That's why most production systems default to a cheaper linear loop and reserve search-based planning for tasks where the extra reliability is worth the extra compute.
Can an agent recover from a mistake without an explicit backtracking mechanism?
Only partially. A ReAct-style loop can adjust once it observes a clearly failed action (a tool error, an unexpected result), because that failure becomes part of its next input. But it has no way to recognize that an earlier, superficially successful step was actually the wrong choice — that requires either explicit backtracking, a verification step, or a human catching it.
If you're deciding which of these planning architectures fits a specific product or workflow, the team at Woyce Technologies can help you scope and build it.