EARTH LINC · Field Note
Field manual · AI systems · 2025现场手册 · AI 系统 · 2025

The engine
is not
the car.
引擎
不等于
整台车

Someone told you: AI model = engine, harness = vehicle. It's a good metaphor. It's also going to mislead you in exactly three places. Here's the teardown — what a harness is, how to build one, how to see one, and where to read next. 有人告诉你:AI 模型 = 引擎,Harness(挂载系统)= 整车。这个比喻很好用,但恰好会在三个地方把你带偏。下面是拆解:Harness 是什么、怎么造、怎么看见它,以及接下来读什么。

Spec plate铭牌
Model模型 = Engine引擎 holds · raw capability, no destination成立 · 只有原始能力,没有方向
HarnessHarness = Vehicle整车 holds — until the harness starts changing the horsepower成立 —— 直到你发现挂载系统本身会改变马力
Mostly true — see §05大致成立 — 见 §05
§00 — Short answer简答

Yes. With one caveat that changes how you build.是的。但有一个会改变你造法的前提

A model converts context into tokens the way an engine converts fuel into torque. Nothing in it steers, brakes, remembers where you were going, or refuses to drive off a cliff. 模型把上下文变成 token,就像引擎把燃料变成扭矩。它本身不会转向、不会刹车、不记得你要去哪,也不会拒绝开下悬崖。

All of that is the harness: the code around the weights that decides what the model sees, what it can touch, when it stops, and how you know whether it worked. In a shipped AI product, the harness is where most of the engineering hours and almost all of the failure modes live. 这些全都归 Harness 管:包在权重外面的那一层代码,决定模型看到什么、能碰什么、何时停下、以及你如何知道它有没有做对。在真正上线的 AI 产品里,绝大多数工时和几乎全部故障,都发生在 Harness 里。

But a chassis doesn't change an engine's torque curve — and a harness absolutely changes a model's measured ability. That single asymmetry is where the metaphor starts to leak. 但底盘不会改变引擎的扭矩曲线 —— 而 Harness 一定会改变模型被测出来的能力。这个不对称,就是比喻开始漏水的地方。

Buy the engine. Build the vehicle. Expect to rebuild it when the engine changes. 引擎靠买,车靠造;引擎一换,车就得重造。
§01 — How do I see a harness?我怎么看见 Harness

Five questions that make it visible五个问题,让它现形

A harness is invisible because it's the part nobody demos. Open any AI product — a coding agent, a support bot, a research assistant — and ask these five. The answers are the harness. Harness 之所以看不见,是因为它是没人拿来演示的那部分。打开任何 AI 产品 —— 编程 agent、客服机器人、研究助手 —— 问下面五个问题。答案本身就是 Harness。

  1. What actually enters the context window this turn? System prompt, retrieved documents, prior turns, tool output, file trees — and what got cut to fit.这一轮到底有什么进了上下文窗口?系统提示、检索到的文档、历史对话、工具输出、文件树 —— 以及为了塞得下被砍掉了什么。
  2. What can it do that isn't producing text? List the tools, their arguments, and their blast radius. That list is the drivetrain.除了输出文字,它还能做什么?列出全部工具、参数和影响范围。这份清单就是传动系统。
  3. Who decides when it stops? A step limit, a budget, a confidence check, a human approval — something must, or it loops forever.谁决定它什么时候停?步数上限、预算、置信度判断、人工确认 —— 总得有一个,否则它会一直循环。
  4. What's written down? Traces, tool calls, token counts, latency, cost. No dashboard, no debugging.有什么被记录下来?轨迹、工具调用、token 数、延迟、成本。没有仪表盘,就没法排障。
  5. How is "it worked" measured? If the answer is "it feels better", there is no test track, and every change is a guess."做对了"是怎么衡量的?如果答案是"感觉变好了",那就没有试车场,每一次改动都是猜。
§02 — Where the word comes from这个词从哪来

Harness is a borrowed word. Twice.Harness 是借来的词,借了两次

It didn't start in AI. In software engineering, a test harness is the code that sets up, feeds, runs and checks a unit under test. AI borrowed it for evaluation harnesses — the machinery that pipes a benchmark into a model and scores the output. Then agent builders borrowed it again for the loop, tools and context that wrap a model in production. Research papers more often call this last one scaffolding. 它不是从 AI 开始的。在软件工程里,test harness(测试挂载)是负责搭建、喂数据、运行并检查被测单元的那段代码。AI 先把它借去做评测 harness——把基准题送进模型、再给输出打分的那套机器。随后 agent 开发者又借了一次,用来指生产环境中包住模型的循环、工具与上下文。论文里更常把最后这一种称作 scaffolding(脚手架)

Keep the three senses separate when you read, or you'll think two papers disagree when they're talking about different objects. 阅读时把这三层含义分清楚,否则你会以为两篇论文互相矛盾,其实它们说的根本不是同一个东西。

§03 — What (anatomy)是什么(构造)

The parts list零件清单

Eight components. If your system is missing three of them, it's not an agent — it's a demo. 八个部件。如果你的系统少了其中三个,那它不是 agent,只是个 demo。

01

Context assembly上下文组装

≈ fuel injection≈ 燃油喷射

Choosing, ordering, compressing and cutting what reaches the model this turn. Most of your quality wins live here, not in prompt wording.选择、排序、压缩、裁剪这一轮送进模型的内容。绝大多数质量提升来自这里,而不是提示词的措辞。

02

Tool interface工具接口

≈ transmission≈ 变速与传动

Turning intent into action: schemas, argument validation, error messages the model can actually recover from. Designed for a model to use, not a human.把意图变成动作:接口定义、参数校验、以及模型真能自我纠正的错误信息。是为模型设计的界面,不是为人。

03

Control loop控制循环

≈ steering + throttle≈ 转向与油门

Plan → act → observe → repeat, plus retries, branching and the stop condition. Fifty lines of code that decide everything.规划 → 执行 → 观察 → 重复,加上重试、分支和终止条件。五十行代码,决定一切。

04

Memory & state记忆与状态

≈ odometer + logbook≈ 里程表与行车记录

Scratchpads, summaries of old turns, durable facts, working files. What survives when the window overflows.草稿区、历史摘要、长期事实、工作文件。当窗口溢出时,还能留下来的东西。

05

Guardrails & permissions护栏与权限

≈ brakes + limiter≈ 刹车与限速器

Sandboxes, allow-lists, spend caps, human approval on irreversible actions. Never rely on the prompt to be the brake.沙箱、白名单、花费上限、不可逆操作的人工确认。永远不要把提示词当刹车用。

06

Observability可观测性

≈ dashboard + OBD port≈ 仪表盘与诊断口

Full traces of every call, token, tool result and cost. Read ten real traces end to end; it beats any amount of theorising.完整记录每一次调用、token、工具结果和成本。把十条真实轨迹从头读到尾,胜过任何空想。

07

Evaluation suite评测集

≈ the test track≈ 试车场

A frozen set of tasks with checkable outcomes. Without it you cannot tell a model upgrade from a harness regression.一组冻结的、结果可判定的任务。没有它,你分不清是模型升级了,还是 Harness 退化了。

08

Cost & latency governor成本与延迟调速

≈ gearbox + fuel gauge≈ 挡位与油量表

Model routing, caching, parallelism, step budgets. Quality per dollar per second is the real spec sheet.模型路由、缓存、并行、步数预算。每一元、每一秒换到多少质量,才是真正的参数表。

§04 — How (build the smallest one)怎么做(先造最小的那台)

Fifty lines, three tools, one scoreboard五十行代码,三个工具,一块记分板

Don't start with a framework. Start with a loop you can hold in your head, on a task where "correct" is machine-checkable — tests pass, the number matches, the file compiles. 别从框架开始。从一个你脑子里装得下的循环开始,选一个"对不对"能被机器判定的任务 —— 测试通过、数字对上、文件能编译。

# the whole idea, minus the yak-shaving
state = {goal, files, notes}

for step in range(MAX_STEPS):          # stop condition = brakes
    ctx  = assemble(state)             # fuel injection — most wins here
    out  = model(ctx, tools)           # the ENGINE
    if out.is_final: break
    if violates(policy, out): halt()   # guardrail, not a prompt
    obs   = run_tool(out.call)         # the drivetrain
    state = update(state, out, obs)    # memory
    log(step, ctx, out, obs, cost)     # dashboard

score(result, rubric)                  # the test track

Pick one checkable task选一个可判定的任务

"Fix this failing test", "extract these 6 fields from this PDF". Vague tasks make ungradeable harnesses.比如"修好这个失败的测试""从这份 PDF 抽出这 6 个字段"。模糊任务只会造出无法打分的 Harness。

Write the loop by hand手写那个循环

No agent library for the first version. You need to feel every decision the framework would have hidden.第一版不要用 agent 库。你需要亲手感受框架会替你藏起来的每一个决定。

Define 3–7 tools定义 3 到 7 个工具

Read, run, write. Make the error messages instructive — a good tool teaches the model how to retry.读、执行、写。让报错信息本身具有指导性 —— 好工具会教模型怎么重试。

Freeze 20–50 eval cases冻结 20 到 50 个评测样例

From your real domain, with expected outcomes. Never edit them to make a run look good.来自你真实的业务领域,附带期望结果。永远不要为了跑分好看去改它们。

Ablate one part at a time一次只拆一个部件

Remove memory. Remove retries. Halve the context. Record the delta. This is the experiment that teaches you the most.去掉记忆。去掉重试。把上下文砍半。记录分差。这是收获最大的实验。

Then swap the engine最后换引擎

Same harness, two models. Compare that gap with your ablation gaps — now you know which half you're actually building.同一个 Harness,两个模型。把这个差距和拆件实验的差距对比 —— 你就知道自己真正在造的是哪一半。

§05 — Where the metaphor breaks比喻在哪里失效

Three caveats三个前提

01

The harness changes the horsepowerHarness 会改变马力

A torque curve belongs to the engine alone. A model's measured ability does not — it is a property of model × scaffold × task. Chain-of-thought prompting, self-consistency sampling and better agent interfaces all move benchmark scores with zero change to the weights. When someone quotes a model score, ask what harness produced it.扭矩曲线只属于引擎。模型被测出的能力却不是 —— 它是"模型 × 脚手架 × 任务"的属性。思维链提示、自洽采样、更好的 agent 接口,都能在权重完全不变的情况下改变基准分数。所以别人报一个模型分数时,先问:是哪套 Harness 跑出来的?

Wei et al., Chain-of-Thought   Wang et al., Self-Consistency   Yang et al., SWE-agent / ACI

02

A new engine can delete your vehicle新引擎会把你的车删掉

Cars don't become obsolete because the engine improved. Harnesses do. Every clever workaround you build to compensate for today's model weakness becomes dead weight when the next model ships — that's Sutton's bitter lesson wearing work clothes. So build thin where the model is improving fast (planning, reasoning, decomposition) and thick where it will never help you (permissions, data access, audit trails, cost control, your product's UI, your domain's ground truth).汽车不会因为引擎变强而报废,Harness 会。你为弥补当下模型缺陷而写的每一个巧妙补丁,等下一代模型发布就变成了负重 —— 这就是穿着工装的"苦涩教训"。所以:在模型进步最快的地方(规划、推理、任务分解)要薄;在模型永远帮不了你的地方(权限、数据访问、审计、成本控制、产品界面、行业事实)要厚。

Sutton, The Bitter Lesson

03

The failures are semantic, not mechanical故障是语义的,不是机械的

A cracked piston fails in bounded, physical ways. A model fails by being confidently wrong, or by obeying an instruction hidden in a web page it just read. Prompt injection has no mechanical analogue — you can't torque-wrench your way out of an adversary who writes English. Brakes here mean sandboxes, least privilege and human approval on irreversible actions, not a stern system prompt.活塞裂了,故障是有界的、物理的。模型的故障是"自信地答错",或者听从了它刚读到的网页里藏着的一条指令。提示注入没有机械对应物 —— 面对一个会写自然语言的对手,扳手是拧不出安全的。这里的刹车是沙箱、最小权限、不可逆操作的人工确认,而不是一段语气严厉的系统提示。

Willison on prompt injection   OWASP Top 10 for LLM Apps

And one loop the metaphor can't hold at all: vehicles don't build engines. But agent harnesses generate the traces, environments and reward signals used to train the next model. The vehicle is quietly reshaping the engine — which is why "just a wrapper" has been wrong for a while now. 还有一个比喻完全装不下的回路:车不会造引擎。但 agent 的 Harness 会产出用于训练下一代模型的轨迹、环境和奖励信号。车正在悄悄重塑引擎 —— 这也是"不过是个套壳"这句话早就不成立的原因。

§06 — Harnesses in the wild真实世界里的 Harness

Go read these six先读这六个

Eval harness评测 harness

lm-evaluation-harness

EleutherAI's framework for scoring language models. The literal harness that produced many numbers you've quoted.EleutherAI 的语言模型评测框架。你引用过的很多分数,就是这套"挂载"跑出来的。

github.com/EleutherAI/lm-evaluation-harness
Agent scaffoldAgent 脚手架

SWE-agent

Shows that designing the interface a model uses to browse and edit code — not the model — drives the score.证明了拉高分数的是模型浏览与修改代码所用的界面设计,而不是模型本身。

arxiv.org/abs/2405.15793
The loop循环范式

ReAct

Thought → action → observation. The 2022 paper that most agent control loops are still a variation of.思考 → 行动 → 观察。2022 年那篇论文,如今多数 agent 控制循环仍是它的变体。

arxiv.org/abs/2210.03629
The benchmark基准

SWE-bench

Real GitHub issues, graded by whether the tests pass. A test track where the harness is measurably half the car.真实的 GitHub issue,以测试是否通过来打分。在这个试车场上,Harness 实打实占了半台车。

swebench.com
Engineering guide工程指南

Building effective agents

Anthropic's argument for simple, composable workflows over elaborate frameworks. Read before you install anything.Anthropic 主张用简单可组合的工作流,而不是复杂框架。装任何库之前先读它。

anthropic.com/engineering/building-effective-agents
Mounting bolts安装接口

Model Context Protocol

An open standard for connecting tools and data to models — the attempt to standardise the engine mounts.把工具与数据接入模型的开放标准 —— 一次把"引擎支架"标准化的尝试。

modelcontextprotocol.io
§07 — Next steps下一步

A four-week training plan一份四周训练计划

Reading gives you the vocabulary. Only a measured ablation gives you the answer.读文献给你词汇,只有做过一次有数据的拆件实验,才能给你答案。

Week 1 · Reproduce第 1 周 · 复现

Run an eval harness yourself亲手跑一次评测 harness

Install lm-evaluation-harness, run one small open model on two tasks. Change the few-shot count and the prompt template. Watch the score move while the weights don't.装上 lm-evaluation-harness,用一个小型开源模型跑两个任务。改一下 few-shot 数量和提示模板。看着权重没变、分数却在动。

Week 2 · Build第 2 周 · 构建

Write the fifty-line loop写出那五十行循环

Three tools: read file, run command in a sandbox, write file. A step limit and a spend cap. No framework, no abstractions you didn't type.三个工具:读文件、在沙箱里执行命令、写文件。加上步数上限和花费上限。不用框架,不用你没亲手敲过的抽象。

Week 3 · Measure第 3 周 · 度量

Thirty cases and an ablation table三十个样例与一张拆件表

Freeze 30 tasks from your own domain with checkable outcomes. Then remove memory, remove retries, halve the context, degrade the tool errors. Record every delta.从自己的领域冻结 30 个结果可判定的任务。然后去掉记忆、去掉重试、把上下文砍半、把工具报错变模糊。每一次的分差都记下来。

Week 4 · Compare第 4 周 · 对比

Swap engines, keep the vehicle换引擎,留下车

Same harness, two different models, same 30 cases. Then read one real agent codebase end to end — SWE-agent is a good first one — and mark which of your eight parts it has.同一个 Harness、两个不同模型、同样 30 个样例。然后把一个真实 agent 代码库从头读到尾(SWE-agent 是不错的第一个),标出它具备八个部件中的哪几个。

Your deliverable: one table. Rows = harness variants. Columns = models. Cells = score, cost, latency. If the spread across rows rivals the spread across columns, you've just proved the metaphor's most important half — and its limit — with your own data.你的交付物:一张表。行 = Harness 变体,列 = 模型,格子 = 分数、成本、延迟。如果行与行之间的差距能和列与列之间的差距相当,你就用自己的数据同时证明了这个比喻最重要的一半,以及它的边界。

§08 — References & further learning参考资料与延伸阅读

Sources出处

Everything asserted above traces back to one of these. Primary sources first; read the papers, not the summaries of the papers.上面每一条判断都能追溯到下面某一项。优先读一手资料 —— 读论文本身,而不是论文的转述。

Origins & vocabulary词源与术语

Loops, scaffolds & tools (papers)循环、脚手架与工具(论文)

Harnesses & benchmarksHarness 与基准

Engineering guides工程指南

Risk, guardrails & brakes风险、护栏与刹车