How AI Agents Actually Work (and Why Most Break)

Okay so “AI agent” is the buzzword of the year. Every company says they are building agents. But what does that actually mean under the hood?

Let me show you. The concepts are pretty simple once you see them. The tricky part is most agents in production are held together with hope and prompt engineering. That’s why they break. Let me walk you through what’s really happening and what it takes to build ones that don’t fall apart.

1. An agent is just a loop

Here is the simplest definition I can give you. An agent is an LLM that can decide to do things, in a loop.

A normal LLM call is one-shot. You send a prompt, you get a response, done. Nothing happens after that.

An agent is different. You give it a task. Then the agent can:

• Think about what to do next
• Call a tool to actually do something (search the web, read a file, call an API)
• Look at the result
• Decide what to do next
• Repeat until the task is done

Analogy: a normal LLM call is like asking a friend a single question. An agent is like asking a friend to go run an errand for you. They figure out the steps, go do them, come back, and tell you the result.

Step through the demo. You will see the loop running. Think, act, observe, think again. Each time the agent has new information, it decides what to do next. This pattern is called a “ReAct loop” in the research papers but it is really just thinking out loud plus doing stuff.

2. Why simple chains break

Before agents became a thing, people would build LLM apps as a chain. Step 1, step 2, step 3, done. Output of one prompt feeds into the next one. Totally linear.

This works fine for simple stuff. “Summarize this, then translate it, then format as JSON.” Three fixed steps. No surprises.

But real business problems are not linear. They branch. They loop. Sometimes they need human input in the middle. Sometimes step 3 should happen before step 2. Sometimes you need to skip step 2 entirely depending on what step 1 returned.

Imagine you want to build a customer support agent. The flow is: “classify the message, then either answer directly, OR look up the customer’s account, OR hand off to a human, OR ask for clarification.” A linear chain cannot do that. You need decisions.

So people started adding if-else statements around chains. Then while loops. Then retry logic. It all works for a while but gets messy fast. What you actually want is a way to say “here is my workflow, each step is a node, and here are the rules for moving from one step to another.”

That is where state machines come in.

3. Agents are really graphs

The best way to think about an agent is as a directed graph. Nodes are steps. Edges are “go here next.” State flows from node to node along the edges.

Each node does one specific thing. Maybe it calls an LLM. Maybe it calls a tool. Maybe it runs some plain code. Maybe it asks the user a question. The node reads the current state, does its thing, updates the state, and hands off to the next node.

Click through the demo. Watch how the same graph routes different inputs through different paths. Some go straight to the answer. Some need research first. Some loop back to ask the user for more info.

This is what every serious agent framework gives you. LangGraph, Pydantic AI, CrewAI, whatever. They all give you a way to define your workflow as a graph with typed state that flows through it. The framework differences are mostly syntax. The core idea is the same.

4. The LLM decides, code routes

Here is the most important pattern for reliable agents. Burn this into your brain.

Let the LLM make judgment calls. Let code handle the routing.

What does that mean?

LLM decides: “this user is asking a technical question, not a billing question.” That is judgment. You cannot hardcode a classifier that works on free-form language.

Code routes: based on the classification, send to the technical support node or the billing node. That is a switch statement. You do not need an LLM for it.

Bad pattern: ask the LLM “what should you do next?” and trust its answer. The LLM might hallucinate a tool name that does not exist. Or skip a required validation step. Or call a dangerous tool when it should have asked for approval.

Good pattern: ask the LLM “what category is this message?” and then use code to pick the next step based on the category. The LLM is constrained to answering the question you asked. Code does the branching and keeps things predictable.

This is the core of building reliable agents. Use LLMs where they are great (judgment, understanding language, being creative). Use code where code is great (rules, flow control, validation). Do not mix them up.

5. How tools actually work

When people say “the agent used a tool” here is what actually happens under the hood.

You tell the LLM: “here are the tools you can call. Tool 1 is called search_web and it takes a query string. Tool 2 is called read_file and it takes a path. Tool 3 is…”

The LLM sees these tool descriptions in its prompt. When it decides to use a tool, the LLM does NOT actually call the tool. It only outputs a special formatted message like this:

{
  "tool": "search_web",
  "arguments": { "query": "weather in karachi" }
}

Then YOUR CODE parses that output, calls the actual function, gets the result, and feeds the result back into the LLM as the next message. The LLM reads the result and decides what to do next.

The LLM never runs any code. It only generates text. Your code does all the actual work. “Tool use” is really just a structured conversation where the LLM asks for something and your code does it.

This matters a lot because it means you have full control. You can log every tool call. You can validate arguments before running anything. You can refuse to call certain tools in certain contexts. You can add rate limits. All of that is YOUR code, not the LLM’s responsibility.

6. Errors are going to happen

Every production agent hits errors. API timeouts. Rate limits. A tool returns unexpected JSON. The LLM outputs something that does not parse. A service is down.

You cannot avoid errors. But you can handle them gracefully.

Retries with backoff: if a tool call fails with a network error, try again after 1 second, then 2, then 4. Most transient errors go away by the third try.

Fallback tools: if the primary search API is down, try the backup. If Claude is rate limited, fall back to a different model.

Graceful degradation: if something really cannot be done, tell the user instead of crashing. “I could not find weather data right now. Here is what I could do instead.”

Checkpointing: save the agent’s state after every step. If something crashes, resume from the last checkpoint instead of starting over. Especially important for long-running tasks that cost money.

Human in the loop: for important decisions (sending an email, charging a card, deleting files), pause the agent and ask a human to confirm. This is not a bug. This is a feature. Users trust agents more when they can see what is about to happen.

7. Memory is harder than it looks

An agent that cannot remember what happened two steps ago is not very useful. But memory is where most agents get stuck.

There are basically two kinds of memory to think about, and they work very differently.

Short-term memory is the context of the current task. Steps taken so far, tools called, results received. This is just conversation history. You keep appending to it.

Problem: conversation history gets long fast. Long context costs money and slows things down. And models have limits (200K, 1M tokens, whatever). Long-running agents will hit those limits.

Long-term memory is stuff the agent should remember across sessions. User preferences, past interactions, important facts. This is NOT just context. You need a separate storage system. A vector database for semantic search. A regular database for structured facts. Files for documents.

The agent reads from long-term memory when it starts a task, and writes to it when something important happens. It does not keep everything in the prompt.

Most people conflate these two and try to dump everything into context. That does not scale. Use the right kind of memory for the right kind of data.

8. Guardrails you actually need

Here are the guardrails every production agent should have. None of these are optional.

Structured outputs. Never trust raw LLM text for anything important. Use Pydantic, Zod, or JSON schemas. Validate every output before acting on it.

Budget limits. Agents can loop forever and burn thousands of dollars. Set a max number of iterations (say 20). Set a max token budget per task. If the agent is not done by then, stop and report.

Tool allowlists. Each node should know exactly which tools it can call. Do not let a classification node accidentally call your “send_email” tool because the LLM went off-script.

Observability. Log every LLM call, every tool call, every state transition, with timestamps and inputs and outputs. When something breaks in production, you need to replay what happened. Without logs you are guessing.

Cost tracking. Tokens by node, by tool, by user. You need to know where your money goes. Surprise $5000 bills happen.

Without these, your agent works great in testing and blows up in production.

9. When to use an agent (and when not to)

Agents are powerful but they are also slow, expensive, and non-deterministic. They are not always the right answer.

Use an agent when:

• The task needs real judgment the user cannot describe upfront
• The path through the workflow depends on data you only get at runtime
• Multiple tools need to be coordinated
• The task is open-ended or exploratory

Do NOT use an agent when:

• The task is deterministic and every step is known upfront. Just write a script.
• Latency matters a lot. A simple LLM call takes 2 seconds. A 5-step agent takes 10 to 15.
• Cost matters and the task is simple. Agents burn way more tokens than single calls.
• Reliability is critical and you cannot accept non-deterministic behavior.

A lot of stuff getting sold as “AI agent” right now is really just an LLM call with a retry. That is fine. Not every problem needs an agent.

So that’s basically it

An agent is just an LLM in a loop that can use tools and make decisions. Modeling the workflow as a state graph keeps it reliable. Letting code handle routing keeps it predictable. Guardrails keep it from burning your money.

The hard part is not the agent framework. The hard part is designing the right graph for your problem, picking the right tools, handling errors gracefully, and knowing when an agent is actually not the right answer.

I have built agentic systems that handle 10,000+ decisions a day for document processing, lead qualification, and customer onboarding. The pattern is always the same. Explicit state management, structured outputs, code-level guardrails around LLM judgment. Boring on the outside, solid on the inside.

If you are building something like this and want a second pair of eyes on the design, get in touch.

Sources

• Building effective agents (Anthropic). Required reading. Anthropic’s own guide, argues for simple composable patterns over heavy frameworks. The best overview you will find on when to use an agent vs a workflow.
• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al.). The paper where the think-act-observe loop comes from. Surprisingly readable for a research paper.
• LangGraph docs. The state graph framework from the LangChain team. Good examples of the “LLM decides, code routes” pattern in practice.
• Pydantic AI docs. Lighter alternative to LangGraph with strong typing via Pydantic models. Good if you care about structured outputs and validation.