{"id":669,"date":"2026-04-05T17:21:16","date_gmt":"2026-04-05T09:21:16","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=669"},"modified":"2026-04-05T17:21:16","modified_gmt":"2026-04-05T09:21:16","slug":"meet-autoagent-the-open-source-library-that-lets-an-ai-engineer-and-optimize-its-own-agent-harness-overnight","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=669","title":{"rendered":"Meet \u2018AutoAgent\u2019: The Open-Source Library That Lets an AI Engineer and Optimize Its Own Agent Harness Overnight"},"content":{"rendered":"

There\u2019s a particular kind of tedium that every AI engineer knows intimately: the prompt-tuning loop. You write a system prompt, run your agent against a benchmark, read the failure traces, tweak the prompt, add a tool, rerun. Repeat this a few dozen times and you might move the needle. It\u2019s grunt work dressed up in Python files. Now, a new open-source library called AutoAgent<\/strong>, built by Kevin Gu at thirdlayer.inc<\/a>, proposes an unsettling alternative \u2014 don\u2019t do that work yourself. Let an AI do it.<\/p>\n

AutoAgent is an open source library for autonomously improving an agent on any domain. In a 24-hour run, it hit #1 on SpreadsheetBench with a score of 96.5%, and achieved the #1 GPT-5 score on TerminalBench with 55.1%.<\/p>\n

\n
\"\"
https:\/\/x.com\/kevingu\/status\/2039843234760073341<\/figcaption><\/figure>\n<\/div>\n

What Is AutoAgent, Really?<\/strong><\/h3>\n

AutoAgent is described as being \u2018like autoresearch but for agent engineering.\u2019 The idea: give an AI agent a task, let it build and iterate on an agent harness autonomously overnight. It modifies the system prompt, tools, agent configuration, and orchestration, runs the benchmark, checks the score, keeps or discards the change, and repeats.<\/p>\n

To understand the analogy: Andrej Karpathy\u2019s autoresearch<\/code> does the same thing for ML training \u2014 it loops through propose-train-evaluate cycles, keeping only changes that improve validation loss. AutoAgent ports that same ratchet loop from ML training into agent engineering. Instead of optimizing a model\u2019s weights or training hyperparameters, it optimizes the harness<\/em> \u2014 the system prompt, tool definitions, routing logic, and orchestration strategy that determine how an agent behaves on a task.<\/p>\n

A harness<\/strong>, in this context, is the scaffolding around an LLM: what system prompt it receives, what tools it can call, how it routes between sub-agents, and how tasks are formatted as inputs. Most agent engineers hand-craft this scaffolding. AutoAgent automates the iteration on that scaffolding itself.<\/p>\n

The Architecture: Two Agents, One File, One Directive<\/strong><\/h3>\n

The GitHub repo<\/a> has a deliberately simple structure. agent.py<\/code> is the entire harness under test in a single file \u2014 it contains config, tool definitions, agent registry, routing\/orchestration, and the Harbor adapter boundary. The adapter section is explicitly marked as fixed; the rest is the primary edit surface for the meta-agent. program.md<\/code> contains instructions for the meta-agent plus the directive (what kind of agent to build), and this is the only file the human edits. <\/p>\n

Think of it as a separation of concerns between human and machine. The human sets the direction<\/em> inside program.md<\/code>. The meta-agent<\/strong> (a separate, higher-level AI) then reads that directive, inspects agent.py<\/code>, runs the benchmark, diagnoses what failed, rewrites the relevant parts of agent.py<\/code>, and repeats. The human never touches agent.py<\/code> directly.<\/p>\n

A critical piece of infrastructure that keeps the loop coherent across iterations is results.tsv<\/code> \u2014 an experiment log automatically created and maintained by the meta-agent. It tracks every experiment run, giving the meta-agent a history to learn from and calibrate what to try next. The full project structure also includes Dockerfile.base<\/code>, an optional .agent\/<\/code> directory for reusable agent workspace artifacts like prompts and skills, a tasks\/<\/code> folder for benchmark payloads (added per benchmark branch), and a jobs\/<\/code> directory for Harbor job outputs.<\/p>\n

The metric is total score produced by the benchmark\u2019s task test suites. The meta-agent hill-climbs on this score. Every experiment produces a numeric score: keep if better, discard if not \u2014 the same loop as autoresearch. <\/p>\n

The Task Format and Harbor Integration<\/strong><\/h3>\n

Benchmarks are expressed as tasks in Harbor format. Each task lives under tasks\/my-task\/<\/code> and includes a task.toml<\/code> for config like timeouts and metadata, an instruction.md<\/code> which is the prompt sent to the agent, a tests\/<\/code> directory with a test.sh<\/code> entry point that writes a score to \/logs\/reward.txt<\/code>, and a test.py<\/code> for verification using either deterministic checks or LLM-as-judge. An environment\/Dockerfile<\/code> defines the task container, and a files\/<\/code> directory holds reference files mounted into the container. Tests write a score between 0.0 and 1.0 to the verifier logs. The meta-agent hill-climbs on this. <\/p>\n

The LLM-as-judge<\/strong> pattern here is worth flagging: instead of only checking answers deterministically (like unit tests), the test suite can use another LLM to evaluate whether the agent\u2019s output is \u2018correct enough.\u2019 This is common in agentic benchmarks where correct answers aren\u2019t reducible to string matching.<\/p>\n

Key Takeaways<\/strong><\/h3>\n