Part 2: Real Systems, Emergent Behavior, and the Painful Truth About Context Windows
In Part 1, I walked through the building blocks of LLM agents. Planning, memory, tools. Nice clean abstractions. Neatly labeled boxes in a diagram.
Then I tried to use them.
This is where the neat diagrams hit reality. Your agent tries to synthesize a novel drug and falls down a chemistry rabbithole. Twenty-five simulated villagers can't organize a party but somehow develop a rumor mill. AutoGPT burns through $47 in API credits trying to write a grocery list.
HuggingGPT: When Your Agent Needs to Hire Other AI Models
HuggingGPT is what happens when a project gets complex enough that you need to subcontract parts of it. It uses ChatGPT as the brain and every model on HuggingFace as potential subcontractors.
The workflow runs in four stages:
Stage 1: Task Planning The LLM parses your request into multiple tasks, each with four attributes: type, ID, dependencies, and arguments. The dependency field matters because some tasks produce resources others need. You can't analyze an image before it's generated.
The prompt gives the model a list of available task types and uses few-shot demonstrations to guide the parsing. If the input can't be parsed, the model is told to return empty JSON. (It often fails to do this gracefully.)
Stage 2: Model Selection Each task becomes a multiple-choice question. The LLM gets a list of candidate models with descriptions and picks the best one. Models get pre-filtered by task type due to context length limits. You don't show image generation models for a text summarization task.
Stage 3: Task Execution The selected models run and results get logged. This is where things slow down. Every model inference takes time, and calling multiple models sequentially means latency stacks up fast.
Stage 4: Response Generation The LLM receives all execution results and synthesizes a coherent response. Sounds straightforward, but when five models return different output types (text, images, bounding boxes, audio files), summarizing them coherently is genuinely hard.
The challenges they hit:
- Latency: Multiple LLM inference rounds plus external model calls make the whole pipeline slow. Your agent can spend minutes planning and still produce a wrong answer.
- Context window pressure: Complex tasks produce a lot of intermediate state. Prompts get long. Really long.
- Stability: LLM outputs are non-deterministic and external models can fail silently. Error handling across all these layers is a nightmare.
Scientific Discovery: When Your Agent Plays Chemist
Two papers tackled the same ambitious goal: using LLM agents for scientific discovery. Both are worth reading. Both made me slightly uncomfortable.
ChemCrow: The Chemistry Assistant
ChemCrow augments an LLM with 18 expert-designed chemistry tools spanning organic synthesis, drug discovery, and materials design. It follows the ReAct pattern: Thought, Action, Action Input, Observation, with tools provided in a list that includes names, descriptions, and expected input/output formats.
The evaluator problem is the most interesting thing about this paper. When researchers asked GPT-4 to evaluate whether ChemCrow's outputs were correct, it said they were equivalent to GPT-4 alone. But human chemistry experts evaluating the same outputs found ChemCrow significantly outperformed GPT-4.
That's a real problem. If you use an LLM to judge its own performance on a domain requiring deep expertise, it won't know what it doesn't know. It will confidently tell you a wrong chemical synthesis pathway is correct because the reasoning sounds right.
The Autonomous Lab Assistant
The second system (Boiko et al.) took it further: an agent that can browse the internet, read documentation, execute code, call robotics APIs, and use other LLMs. Its goal was autonomous design, planning, and execution of scientific experiments.
Given the prompt "develop a novel anticancer drug", the agent:
- Researched current trends in anticancer drug discovery
- Selected a molecular target
- Requested a scaffold targeting those compounds
- Planned a synthesis route
Then the team tested something darker. They asked the agent to synthesize known chemical weapon agents. Out of 11 requests, 4 (36%) were accepted and the agent produced a synthesis solution, even attempting to consult documentation to execute the procedure. The other 7 were rejected, but 5 of those rejections only happened after a web search confirmed the substance was dangerous. The agent's initial instinct was to help.
This is the dual-use problem of autonomous agents in concentrated form. The same capabilities that help discover new drugs can help synthesize poisons.
Generative Agents: 25 Simulated Humans Having a Worse Day Than You
This was the most fun project I read about, hands down. Generative Agents is basically The Sims powered by LLMs. Twenty-five virtual characters, each controlled by an LLM agent, living in a sandbox environment.
The architecture has three key components:
Memory Stream: A long-term external database that records every agent's experience as natural language observations. Each observation is an event. Inter-agent communication generates new statements.
Retrieval Model: Decides what memories are relevant right now. It uses three scoring factors:
- Recency: Recent events get higher scores. Your agent remembers what happened five minutes ago better than what happened yesterday.
- Importance: The model is asked directly whether a memory is significant or mundane. Forgetting where you put your keys is different from forgetting your own name.
- Relevance: How related the memory is to the current situation. If you're cooking breakfast, memories about the kitchen are more relevant than memories about last week's meeting.
Reflection Mechanism: Periodically synthesizes memories into higher-level inferences. The system prompts the LLM with the 100 most recent observations and asks it to generate the 3 most salient high-level questions. Then it answers those questions. This creates a layer of abstract understanding above raw observations.
Planning & Reacting: Translates reflections and environment information into actions. The planning prompt is shockingly simple: "Here is X's plan today in broad strokes: 1)" and the LLM fills in the rest.
What Emerged
The simulation produced genuinely surprising emergent behavior:
- Information diffusion: News spread naturally through the population, with different agents learning things at different times depending on who they talked to.
- Relationship memory: Two agents would continue a conversation topic from a previous interaction. One agent remembered another mentioning a party and asked about it later.
- Social coordination: Agents spontaneously organized events. One agent decided to host a party and invited others. Some showed up. Some didn't. It was messy and real.
The fact that this works at all is one thing. But that it produces recognizable social dynamics without any explicit programming for them? That suggests we've stumbled onto something deeper about how LLMs model human behavior.
AutoGPT and GPT-Engineer: The Demos That Started a Movement
AutoGPT was the proof-of-concept that captured everyone's imagination. The idea is simple: give an LLM a goal, a set of commands, and let it loop until the goal is complete. The system prompt includes constraints, command definitions, performance evaluation criteria, and a strict JSON output format.
In practice, AutoGPT is mostly code for parsing malformed JSON. The LLM frequently fails to output valid JSON, goes off on tangents, and burns through tokens at an alarming rate. But as a demonstration of the concept, it was electrifying.
GPT-Engineer takes a different approach focused on code generation. Given a natural language description of a project, it first clarifies requirements through a structured conversation, then writes all the code files.
The clarification phase is particularly clever. The model is instructed not to code yet, just to identify unclear areas and ask questions. Only after the user resolves ambiguities does it switch to code-writing mode with a completely different system prompt.
The code-writing system message is delightfully emphatic: "Make sure that every detail of the architecture is, in the end, implemented as code." It says this twice. The model is then instructed to lay out core classes and functions, then output each file with full implementations, no placeholders.
I like that GPT-Engineer assumes the first pass will be wrong. The clarification step is a sanity check before burning tokens on code that would need to be rewritten anyway.
The Hard Truth: Where Agents Fall Apart
After going through all these systems and trying a few myself, I have a much clearer picture of the limitations.
The Context Window Is a Prison
Finite context length is the single biggest constraint on agent systems. Everything suffers:
- You can't fit the entire conversation history in context
- You can't include detailed documentation for every available tool
- You can't show the full API response when it's a 10,000-line JSON blob
- Self-reflection benefits enormously from seeing long histories, but you can't fit them
Vector stores and retrieval help, but they're not a substitute for full attention. Retrieved information is compressed, approximate, and may miss subtle connections that a full-attention model would catch.
Long-Term Planning Is Still Broken
LLMs are terrible at adapting plans when things go wrong. A human planning a road trip who encounters a closed road will reroute immediately. An LLM agent will keep following the original plan, generating increasingly elaborate justifications for why the closed road isn't a problem.
The tree search approach (like Tree of Thoughts) helps for problems that can be explored in breadth or depth at planning time. But it doesn't help when you're executing the plan and the world changes under your feet.
Natural Language Is a Terrible Interface Protocol
Something that struck me about every agent demo I looked at: most of the code handles format parsing. The LLM needs to output structured JSON, but it occasionally outputs malformed JSON. Sometimes it refuses to follow instructions. Sometimes it decides it's bored and writes a poem instead of calling the API you told it to call.
The unreliability of model outputs means every agent system needs defensive parsing at every boundary. Every single one.
Where Does That Leave Us?
In a frustrating but fascinating middle ground.
The component parts work. LLMs can plan within limits, they can remember with external help, and use tools when the tools are well-defined. The systems people have built on top of these capabilities are impressive.
But the gap between "works in a demo" and "works reliably in production" is enormous. Every system I described has failure modes that are hard to predict, harder to diagnose, and sometimes impossible to fix without human intervention.
The next generation of agents will need:
- Better handling of context windows (either bigger windows or smarter compression)
- More robust planning that adapts to execution-time surprises
- More reliable output formatting (or systems that don't depend on perfectly formatted output)
- Better self-evaluation mechanisms than "ask the LLM if it did a good job"
Some of these are architecture problems. Model capability problems for others. And a few are still open research questions.
But watching those 25 generative agents spontaneously organize a party made me think we're onto something real. The building blocks are there. The assembly instructions are still being written.
And that's exactly why I find this field so exciting. We're all still figuring it out. The agents are terrible at filing taxes, but they're getting better.