I Tried Building an LLM Agent and Accidentally Gave It a To-Do List It Could Never Finish

Part 1: The Building Blocks (Planning, Memory, Tool Use)

I sat down, fired up my terminal, and decided to build an AI agent. The kind that plans, remembers things, and maybe files my taxes. (It did not file my taxes.)

Building an LLM-powered autonomous agent is less like assembling IKEA furniture and more like teaching a golden retriever to play chess. The dog is enthusiastic. It understands some concepts. But it keeps trying to eat the pieces.

So Here's the Basic Setup

Think of an LLM-powered agent as a regular language model with three superpowers bolted on. The LLM is the brain. It decides what to do. But the brain alone is useless without:

1. Planning - The ability to break big problems into smaller steps and learn from screw-ups
2. Memory - Remembering what happened five minutes ago (short-term) and what happened last week (long-term)
3. Tool Use - The ability to call external APIs because LLMs are really bad at math and have no idea what the weather is today

Each of these is its own rabbit hole. I fell into all of them.

Planning: Teaching an LLM to Think Ahead

Complex tasks have steps. Lots of them. And LLMs, left to their own devices, will just start generating tokens and hope for the best. That works great for writing a haiku. It does not work great for "plan a week-long vacation itinerary across three countries."

The "Think Step by Step" Trick

The simplest trick in the book is Chain of Thought prompting. You literally tell the model "think step by step" and suddenly its performance on complex reasoning tasks jumps. It sounds ridiculous that this works, but it does. The model uses more compute at inference time to decompose a hard problem into manageable chunks.

Then came Tree of Thoughts, which extends this idea by exploring multiple reasoning paths at each step. Instead of one chain, you get a whole tree of possibilities. The search process uses breadth-first or depth-first search, with each node evaluated by a classifier or majority vote. It treats the LLM's reasoning as a search problem. That is both smart and a little unnerving.

The PDDL Detour

One approach I found particularly amusing is called LLM+P. The idea: instead of making the LLM plan everything itself, outsource the planning to an actual classical planner. Use PDDL (Planning Domain Definition Language) as a middleman. The LLM translates your problem into PDDL, passes it to a classical planner, gets a plan back, and translates it to natural language.

This works great if you have a domain-specific PDDL planner handy. Which, let's be honest, most of us don't. But in robotics and certain structured domains, this is actually viable. The LLM stops being the planner and becomes a translator between human language and formal planning languages.

Learning From Your Mistakes (The Hard Way)

The ability to reflect on past actions and improve is what separates a toy demo from something that might actually work in production. At least some of the time.

ReAct (Reasoning + Acting) is one of those ideas that feels obvious in hindsight. Instead of having the LLM just reason OR just act, you interleave them. The format looks like:

Thought: I need to find out when the Beatles formed.
Action: Search Wikipedia for "The Beatles"
Observation: The Beatles were an English rock band formed in Liverpool in 1960.
Thought: Okay, so they formed in 1960. But when did they break up?
Action: Search Wikipedia for "The Beatles breakup"

The reasoning traces (the "Thought" steps) help the model handle unexpected situations, while the actions let it interact with the world. In the paper's experiments, adding the reasoning traces consistently outperformed acting alone.

Reflexion takes this further by giving the agent dynamic memory of its past failures. It works like a reinforcement learning setup: the agent tries something, gets a reward (usually just binary: success or failure), and if it fails, it reflects on why and stores that reflection for next time.

The reflection is generated by showing the LLM examples of failed trajectories paired with ideal reflections, then asking it to produce its own. These reflections get stored in working memory (up to about three at a time) and used as context for future queries.

The heuristic function that decides when a trajectory is hopeless looks for two things: inefficient planning (taking too long without succeeding) and hallucination (getting stuck repeating the same action over and over). When either is detected, it resets and tries again with the new reflection.

Chain of Hindsight (CoH) takes a different approach. Instead of learning from its own failures at inference time, it's trained on sequences of past outputs annotated with human feedback. The model sees a history like: "Here's what I generated, here's the feedback, here's an improved version." It learns to follow the trend toward better outputs.

The trick to making CoH work is adding a regularization term so the model doesn't overfit to the feedback sequences, and randomly masking some tokens during training to prevent shortcutting. Because if you give an LLM a sequence of "bad, bad, good" outputs, it will figure out that the last one is always the best and just copy it without learning anything.

Algorithm Distillation (AD) applies similar thinking to reinforcement learning tasks. Instead of learning a specific policy, it learns the process of learning itself. The model is fed multi-episode histories (2-4 episodes) that show the agent gradually improving over time. The goal is that when you feed it a new task, it knows how to get better at it episode by episode, even without explicit training on that task.

The results are solid. AD approaches the performance of RL^2 (which requires online RL and is considered an upper bound) while only using offline data. It learns much faster than expert distillation, which just clones expert behavior without understanding the learning process.

Memory: Why Your Agent Needs a Brain and a Notebook

Human memory isn't one thing. It's three things stacked in a trench coat.

The Three Kinds of Memory (Borrowed From Biology)

1. Sensory Memory - Lasts a few seconds. It's the after-image you see when you close your eyes. In LLM terms, this maps to embedding representations of raw inputs.

2. Short-Term / Working Memory - The 7-ish items you can hold in your head at once. This maps directly to in-context learning. Everything in the LLM's context window is short-term memory. And like human short-term memory, it's limited.

3. Long-Term Memory - The stuff you remember from years ago. For LLMs, this is an external vector store that the agent queries when it needs to recall something.

The mapping isn't perfect, but it's useful enough to guide design decisions.

Vector Search: Finding a Needle in a Stack of Needles

For long-term memory, you need a vector database that supports fast Maximum Inner Product Search (MIPS). The idea is simple: convert everything to embeddings, store them, and when you need to remember something, find the closest vectors.

The problem is that brute-force search over millions of vectors is slow. Really slow. So you reach for Approximate Nearest Neighbors (ANN) algorithms, trading a small amount of accuracy for speed. Here are the contenders:

• LSH (Locality-Sensitive Hashing): Uses hash functions that map similar items to the same buckets. Efficient, but the quality depends heavily on the hash function design.

• ANNOY (Approximate Nearest Neighbors Oh Yeah): Builds random projection trees, where each node splits the space with a hyperplane. Search happens across all trees, following the closest half at each step.

• HNSW (Hierarchical Navigable Small World): Inspired by the "six degrees of separation" idea. Builds hierarchical layers of small-world graphs, where top layers provide shortcuts across large distances in the data space. Each move in an upper layer can skip over huge swaths of data.

• FAISS (Facebook AI Similarity Search): Uses vector quantization. Partitions the vector space into clusters, first finds the right cluster (coarse), then searches within it (fine).

• ScaNN (Scalable Nearest Neighbors): Google's entry. The innovation is anisotropic vector quantization, which optimizes quantization to preserve the inner product ranking rather than minimizing reconstruction error.

Each of these trades off between speed, memory, and accuracy. For most agent applications, HNSW or FAISS hit the sweet spot. They give you fast retrieval with good recall, which is exactly what you need when your agent is trying to remember what it was doing three hundred steps ago.

Tool Use: Because LLMs Are Useless at Math

LLMs are terrible at arithmetic, current events, database lookups, and just about any specialized computation.

The solution is obvious: give them tools.

The Router Pattern

MRKL (Modular Reasoning, Knowledge and Language) introduced the idea of a neuro-symbolic architecture where the LLM acts as a router. It receives a query and decides which "expert module" to pass it to. These modules can be neural (other models) or symbolic (calculators, APIs, databases).

Their experiments showed that knowing when to use a tool and how to use it is the hard part. They tested fine-tuning an LLM to call a calculator, and even that simple task was unreliable. The model would fail to extract the right arguments for basic arithmetic. If getting a calculator call right is hard, imagine the complexity of choosing between 50 different APIs.

Teaching Models to Use APIs

TALM (Tool Augmented Language Models) and Toolformer both fine-tune LMs to use external APIs. The dataset is expanded by checking whether adding an API call annotation improves the quality of model outputs. If the API call makes the output better, the example gets kept. If not, it doesn't.

The modern version of this is function calling (OpenAI API), where the model can output structured function calls that the application executes and returns results for.

API-Bank: The Benchmark Nobody Asked For But Everyone Needed

API-Bank is a benchmark for evaluating tool-augmented LLMs. It contains 53 commonly used APIs (search engines, calculators, calendar queries, smart home control, health data, account auth) and 264 annotated dialogues involving 568 API calls.

The workflow is multi-step:

1. Determine if an API call is needed at all
2. Find the right API (they provide an API search engine because 53 APIs is too many to fit in context)
3. Make the call and handle the response
4. Iterate if the result isn't good enough

The benchmark evaluates at three levels:

• Level 1: Can the model call a given API correctly?
• Level 2: Can the model find the right API by searching?
• Level 3: Can the model plan multiple API calls to solve complex requests like booking a trip?

Level 3 is where things fall apart for most models. Coordinating multiple dependent API calls with partial information from each step is hard. I've seen my own agents get stuck in loops trying to book a restaurant, find directions, and check the weather, each step revealing something that contradicts the previous assumption.

What I Learned So Far

Building an agent means giving an LLM three capabilities it doesn't naturally have: planning, memory, and tools. The ability to break things down and learn from your screw-ups. Memory that persists across sessions. And a way to call external systems when the model's weights aren't enough.

Each of these is a deep research area on its own. The combination is powerful. But deploying these things in the real world is where the wheels come off.

That's for Part 2.