Shipping reliable AI apps, part 6 of 7

What an Agent Harness Is, and a Tiny One You Can Build

An agent is a model plus a harness: the loop, tools, stop rules, context trimming and permissions around it. Here's each part, and a tiny harness in Python.

10 min · 2026-10-29 · Watch on YouTube · Subscribe

After this video

You can write a small agent harness yourself: a tool loop with a turn cap, trimmed tool output and allow/ask/deny permissions.

The short version

An agent is a model plus a harness. The model only reads text and writes text; the harness is everything else: the loop that runs the tools the model asks for, the tool definitions it reads, the rules that stop it, the work that keeps its context small, and the permissions that decide what it may do. After this you can build a small one: under twenty lines for the loop, a few tools with clear descriptions and useful errors, a turn cap, trimmed tool output, and allow/ask/deny rules.

Model vs harness

A model takes text in and gives text out. Out of the box it can't run code or keep state between calls. Everything that isn't the model, from the system prompt and tools to the loop and the limits, is the harness. Agent = model + harness.

The plan that did nothing. Ask a bare model to fix a failing test and it writes a plan: read the file, run the tests, patch line 12. Nothing happens until a harness runs those steps.

The agent loop

The model never runs a tool: it asks for one, with stop_reason tool_use. Your code runs the tool, sends back a tool_result with the request's tool_use_id, and calls the model again with the whole conversation, since the Messages API keeps no state. The loop ends when the model answers with no tool call.

A harness loop in Python. The whole engine, with a turn cap and the permission check. client, MODEL, TOOLS, run_tool and allowed are yours.

Tools

A tool is a name, a description and an input schema in JSON Schema. The description is the most important part: say when to use the tool, what it returns and what it won't do. When a tool fails, send the error back as the tool result with is_error true, and make the message actionable so the model can fix its own call.

A run_tests tool. One string input, a description that says when to call it and what comes back.

Stop rules

The natural stop is a reply without a tool call. A confused model can keep calling tools, and agents cost more as their mistakes compound, so add a hard cap on turns. OpenAI's Agents SDK has it built in as max_turns, raising MaxTurnsExceeded.

Cap and hand over. Start with about 20 turns for a small coding task, log every run that hits the cap, and hand those runs to a human instead of retrying blindly.

Context

Every turn adds to the conversation and the whole thing is sent on every call. As context grows, models recall it less accurately (context rot). Keep it small: trim tool output, clear old tool results, and compact near the context limit (a short summary: the goal, what's fixed, what's left) by summarizing and starting a fresh window.

Trim the test log. Return only the last 50 lines of pytest output.

Permissions

Tool results come from outside your app and can carry injected instructions, so the harness decides what runs, not the model. Sort tools into allow, ask and deny; Claude Code's rules check deny first, then ask, then allow. Test in a sandbox first. A denied call still gets a tool result (is_error true, "denied by the user"): every tool request needs its result, or the next call fails.

Allow, ask, deny. Reads and tests run, edits ask you first, anything else is denied. The loop calls allowed() before every tool.

Is the harness dead?

In his AI Engineer talk, Jediah Katz argues fancy harness tricks no longer add much coding intelligence because models learn those skills, but the harness matters more than ever for what models can't absorb: infrastructure, trust and security, and efficiency.

Thin prompts, solid plumbing. Keep clever prompting thin and spend your effort on the loop, the limits, the context and the permissions.

The code

The loop from the video

def run(messages, max_turns=20):
    for turn in range(max_turns):
        reply = call_model(messages)
        messages.append(reply.message)
        if reply.stop_reason != "tool_use":
            return reply.text
        results = []
        for call in reply.tool_calls:
            results.append(run_tool(call))
        messages.append(results)
    raise RuntimeError("hit max_turns")

call_model and run_tool stand in for your API client and your tools.

Permissions from the video

RULES = {"read_file": "allow",
    "run_tests": "allow",
    "edit_file": "ask"}
rule = RULES.get(name, "deny")

Anything not listed is denied.

The whole tiny harness in TypeScript: loop, stop rules, trimming and permissions (92 lines)

// A tiny agent harness: everything around the model that turns its text into actions.
// The model never runs a tool. It asks for one; this loop runs it and sends the result back.

export type ToolUse = { type: "tool_use"; id: string; name: string; input: Record<string, unknown> };
export type ToolResult = { type: "tool_result"; tool_use_id: string; content: string; is_error?: boolean };
export type Text = { type: "text"; text: string };

export type Message =
  | { role: "user"; content: string | ToolResult[] }
  | { role: "assistant"; content: (Text | ToolUse)[] };

export type ModelReply = { stop_reason: "tool_use" | "end_turn"; content: (Text | ToolUse)[] };

// Anything that takes the conversation and the tool list and returns the next reply: a real API client or a script.
export type Model = (messages: Message[], tools: ToolSpec[]) => Promise<ModelReply>;

// What the model reads. Say when to use the tool, what it returns and what it won't do.
export type ToolSpec = { name: string; description: string; input_schema: Record<string, unknown> };
export type Tool = ToolSpec & { run: (input: Record<string, unknown>) => Promise<string> };

export type Rule = "allow" | "ask" | "deny";
export type Permissions = { allow: string[]; ask: string[]; deny: string[] };

// Deny is checked first, then ask, then allow, so a deny beats a narrower allow. Unlisted tools are denied.
export function decide(permissions: Permissions, tool: string): Rule {
  if (permissions.deny.includes(tool)) return "deny";
  if (permissions.ask.includes(tool)) return "ask";
  if (permissions.allow.includes(tool)) return "allow";
  return "deny";
}

// Keep the end of a long tool output, where test failures and errors are.
export function trim(output: string, lines = 50): string {
  const all = output.split("\n");
  return all.length <= lines ? output : all.slice(-lines).join("\n");
}

export type RunOptions = {
  model: Model;
  tools: Tool[];
  permissions: Permissions;
  confirm: (tool: string, input: Record<string, unknown>) => Promise<boolean>; // for "ask": a person types yes or no
  maxTurns?: number;
  outputLines?: number;
};

export type RunResult =
  | { stopped: "answer"; answer: string; turns: number; messages: Message[] }
  | { stopped: "max_turns"; turns: number; messages: Message[] };

export async function run(task: string, o: RunOptions): Promise<RunResult> {
  const maxTurns = o.maxTurns ?? 20;
  const specs: ToolSpec[] = o.tools.map(({ name, description, input_schema }) => ({ name, description, input_schema }));
  const messages: Message[] = [{ role: "user", content: task }];

  for (let turn = 1; turn <= maxTurns; turn++) {
    const reply = await o.model(messages, specs);
    messages.push({ role: "assistant", content: reply.content });

    // The natural stop: plain text, no tool call.
    if (reply.stop_reason !== "tool_use") {
      const answer = reply.content.flatMap((c) => (c.type === "text" ? [c.text] : [])).join("\n");
      return { stopped: "answer", answer, turns: turn, messages };
    }

    const results: ToolResult[] = [];
    for (const call of reply.content) {
      if (call.type !== "tool_use") continue;
      results.push(await runTool(call, o));
    }
    messages.push({ role: "user", content: results });
  }
  // The hard stop: past the cap, a person looks at the run.
  return { stopped: "max_turns", turns: maxTurns, messages };
}

async function runTool(call: ToolUse, o: RunOptions): Promise<ToolResult> {
  const result = (content: string, is_error = false): ToolResult => ({ type: "tool_result", tool_use_id: call.id, content, ...(is_error && { is_error }) });
  const tool = o.tools.find((t) => t.name === call.name);
  if (!tool) return result(`No tool called ${call.name}. Available: ${o.tools.map((t) => t.name).join(", ")}.`, true);

  const rule = decide(o.permissions, call.name);
  if (rule === "deny") return result(`${call.name} is not allowed in this harness.`, true);
  if (rule === "ask" && !(await o.confirm(call.name, call.input))) return result(`The user declined ${call.name}.`, true);

  try {
    return result(trim(await tool.run(call.input), o.outputLines));
  } catch (e) {
    // A useful error the model can act on, not a crash.
    return result(`${call.name} failed: ${e instanceof Error ? e.message : String(e)}`, true);
  }
}

Plug in any model function that takes the messages and tools and returns the next reply. Tested with a scripted model, no API key needed.

Check yourself

The model replies with stop reason tool use. Who runs the tool, and what goes back?
Your code (the harness) runs it and sends back a tool result tagged with the request's tool_use_id; then it calls the model again with the whole conversation.
Name the two ways a harness loop stops.
The natural stop, when the model answers in plain text with no tool call, and a hard cap on turns (max turns), after which a human looks at the run.
Your agent gets worse on turn 30 of a long task. What three things can the harness do about its context?
Trim tool output, clear old tool results it no longer needs, and compact: summarize the run and start a fresh context window with the summary.

Sources

Every claim in this video was checked against these pages. Open one to see the claims it backs and the exact words they rest on.

blog.langchain.com/the-anatomy-of-an-agent-harness 7 claims
- Agent = model + harness; harness engineering builds systems around models.
  agent = model + harness. harness engineering is how we build systems around models to turn them into work engines.
- The harness is every piece of code, configuration and execution logic that isn't the model.
  if you're not the model, you're the harness. a harness is every piece of code, configuration, and execution logic that isn't the model itself.
- A model takes in data (mostly text) and outputs text.
  models (mostly) take in data like text, images, audio, video and they output text. that's it.
- Out of the box a model cannot keep durable state across interactions or execute code.
  out of the box they cannot: maintain durable state across interactions execute code
- A harness includes the system prompt and the tools with their descriptions.
  concretely, a harness includes things like: system prompts tools, skills, mcps + and their descriptions
- A chat product wraps the model in a while loop that tracks previous messages.
  we wrap the model in a while loop to track previous messages and append new user messages.
- A raw model becomes an agent when a harness gives it state, tool execution, feedback loops and enforceable constraints.
  a raw model is not an agent. but it becomes one when a harness gives it things like state, tool execution, feedback loops, and enforceable constraints.
anthropic.com/…/building-effective-agents 5 claims
- Agents are typically LLMs using tools based on environmental feedback in a loop.
  they are typically just llms using tools based on environmental feedback in a loop.
- It is common to add stopping conditions such as a maximum number of iterations.
  it's also common to include stopping conditions (such as a maximum number of iterations) to maintain control.
- Agents' autonomy means higher costs and the potential for compounding errors.
  the autonomous nature of agents means higher costs, and the potential for compounding errors.
- Anthropic recommends extensive testing of agents in sandboxed environments with guardrails.
  we recommend extensive testing in sandboxed environments, along with the appropriate guardrails.
- Keep agent design simple; reduce abstraction layers and build with basic components in production.
  frameworks can help you get started quickly, but don't hesitate to reduce abstraction layers and build with basic components as you move to production.
docs.claude.com/…/working-with-messages 1 claim
- The Messages API is stateless: you send the full conversation history on every call.
  the messages api is stateless, which means that you always send the full conversational history to the api.
docs.claude.com/…/overview 1 claim
- For client tools, the model replies with stop_reason tool_use and tool_use blocks; your code runs the tool and sends back a tool_result.
  claude responds with stop_reason: "tool_use" and one or more tool_use blocks. your code runs the operation and sends back a tool_result
docs.claude.com/…/handle-tool-calls 4 claims
- Each tool_result carries the tool_use_id of the request it answers.
  tool_use_id : the id of the tool use request this is a result for.
- A failed tool can be reported back as the tool_result content with is_error: true.
  you can return the error message in the content along with "is_error": true
- Tool results carry content from outside your control (webpages, email, uploads); treat it as untrusted, it may carry injected instructions.
  tool results often carry content from sources outside your control: webpages, inbound email, user uploads, third-party apis. treat that content as untrusted
- Every tool_use block needs a matching tool_result in the next message, or the API returns an error.
  if you receive an error such as "tool_use ids were found without tool_result blocks immediately after", check that your tool results are formatted correctly.
openai.github.io/…/running_agents 3 claims
- Agent loop: if the model produces tool calls, run them, append the results and re-run the loop.
  if the llm produces tool calls, we run those tool calls, append the results, and re-run the loop.
- The loop ends with a final output when the model produces text and no tool calls.
  the rule for whether the llm output is considered as a "final output" is that it produces text output with the desired type, and there are no tool calls.
- OpenAI Agents SDK raises MaxTurnsExceeded when a run exceeds max_turns.
  if we exceed the max_turns passed, we raise a maxturnsexceeded exception.
docs.claude.com/…/implement-tool-use 2 claims
- A tool definition has a name, a description and an input_schema, a JSON Schema object.
  input_schema a json schema object defining the expected parameters for the tool.
- Detailed tool descriptions are by far the most important factor in tool performance (Anthropic docs).
  provide extremely detailed descriptions. this is by far the most important factor in tool performance.
anthropic.com/…/writing-tools-for-agents 2 claims
- Error responses should give specific, actionable improvements rather than opaque error codes.
  prompt-engineer your error responses to clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks.
- Use pagination, filtering or truncation with sensible defaults for tool responses that could use a lot of context.
  we suggest implementing some combination of pagination, range selection, filtering, and/or truncation with sensible default parameter values for any tool responses that could use up lots of context.
openai.github.io/…/run 1 claim
- In the OpenAI Agents SDK a turn is one AI invocation including its tool calls.
  a turn is defined as one ai invocation (including any tool calls that might occur).
anthropic.com/…/effective-context-engineering-for-ai-agents 4 claims
- Context rot: as tokens in the context window increase, the model's recall of that context decreases.
  as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases.
- Context rot emerges across all models, some degrading more gently.
  while some models exhibit more gentle degradation than others, this characteristic emerges across all models.
- Tool result clearing (dropping old raw tool results) is one of the safest, lightest forms of compaction.
  one of the safest lightest touch forms of compaction is tool result clearing
- Compaction: summarize a conversation nearing the context limit and restart a new context window with the summary.
  compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary.
research.trychroma.com/context-rot 2 claims
- Chroma's context rot report evaluated 18 LLMs.
  in this report, we evaluate 18 llms
- Chroma found model performance degrades as input length increases, even on simple tasks.
  model performance degrades as input length increases, often in surprising and non-uniform ways.
code.claude.com/…/mcp 1 claim
- Claude Code limits MCP tool output to 25,000 tokens by default.
  claude code displays a warning when mcp tool output exceeds 10,000 tokens and limits output to 25,000 tokens by default.
anthropic.com/…/effective-harnesses-for-long-running-agents 1 claim
- Anthropic's long-running harness keeps a claude-progress.txt log of what agents have done.
  a claude-progress.txt file that keeps a log of what agents have done
code.claude.com/…/permissions 2 claims
- Claude Code has allow, ask and deny permission rules; ask rules prompt for confirmation.
  ask rules prompt for confirmation whenever claude code tries to use the specified tool. deny rules prevent claude code from using the specified tool.
- Claude Code evaluates deny, then ask, then allow; the first match wins, so a deny beats a narrower allow.
  rules are evaluated in order: deny, then ask, then allow. the first match in that order determines the outcome
youtube.com/watch 3 claims
- Katz's talk argues fancy harnesses no longer add coding intelligence: models learn the skills harnesses used to patch in.
  He shows why fancy harnesses no longer add coding intelligence: the data flywheel teaches models the skills harnesses used to patch in.
- Katz argues the harness matters more than ever, owning what models can't absorb: infrastructure, trust and security, efficiency.
  Then he argues the harness matters more than ever. It opens new frontiers like Grokbot, and it owns what models can't absorb: infrastructure, trust and security, and efficiency.
- 'Is the Agent Harness Dead?' is a talk by Jediah Katz on the AI Engineer channel (published 19 Oct 2026).
  Is the agent harness dead? Jediah Katz (SpaceXAI, formerly Cursor’s longest-tenured agent-harness engineer) explains what smarter models mean for coding agents

Spotted a mistake? Say so in the video's comments. Fixes are listed on corrections.