Yes. With one caveat that changes how you build.是的。但有一个会改变你造法的前提
A model converts context into tokens the way an engine converts fuel into torque. Nothing in it steers, brakes, remembers where you were going, or refuses to drive off a cliff. 模型把上下文变成 token,就像引擎把燃料变成扭矩。它本身不会转向、不会刹车、不记得你要去哪,也不会拒绝开下悬崖。
All of that is the harness: the code around the weights that decides what the model sees, what it can touch, when it stops, and how you know whether it worked. In a shipped AI product, the harness is where most of the engineering hours and almost all of the failure modes live. 这些全都归 Harness 管:包在权重外面的那一层代码,决定模型看到什么、能碰什么、何时停下、以及你如何知道它有没有做对。在真正上线的 AI 产品里,绝大多数工时和几乎全部故障,都发生在 Harness 里。
But a chassis doesn't change an engine's torque curve — and a harness absolutely changes a model's measured ability. That single asymmetry is where the metaphor starts to leak. 但底盘不会改变引擎的扭矩曲线 —— 而 Harness 一定会改变模型被测出来的能力。这个不对称,就是比喻开始漏水的地方。
Buy the engine. Build the vehicle. Expect to rebuild it when the engine changes. 引擎靠买,车靠造;引擎一换,车就得重造。
Five questions that make it visible五个问题,让它现形
A harness is invisible because it's the part nobody demos. Open any AI product — a coding agent, a support bot, a research assistant — and ask these five. The answers are the harness. Harness 之所以看不见,是因为它是没人拿来演示的那部分。打开任何 AI 产品 —— 编程 agent、客服机器人、研究助手 —— 问下面五个问题。答案本身就是 Harness。
- What actually enters the context window this turn? System prompt, retrieved documents, prior turns, tool output, file trees — and what got cut to fit.这一轮到底有什么进了上下文窗口?系统提示、检索到的文档、历史对话、工具输出、文件树 —— 以及为了塞得下被砍掉了什么。
- What can it do that isn't producing text? List the tools, their arguments, and their blast radius. That list is the drivetrain.除了输出文字,它还能做什么?列出全部工具、参数和影响范围。这份清单就是传动系统。
- Who decides when it stops? A step limit, a budget, a confidence check, a human approval — something must, or it loops forever.谁决定它什么时候停?步数上限、预算、置信度判断、人工确认 —— 总得有一个,否则它会一直循环。
- What's written down? Traces, tool calls, token counts, latency, cost. No dashboard, no debugging.有什么被记录下来?轨迹、工具调用、token 数、延迟、成本。没有仪表盘,就没法排障。
- How is "it worked" measured? If the answer is "it feels better", there is no test track, and every change is a guess."做对了"是怎么衡量的?如果答案是"感觉变好了",那就没有试车场,每一次改动都是猜。
Harness is a borrowed word. Twice.Harness 是借来的词,借了两次
It didn't start in AI. In software engineering, a test harness is the code that sets up, feeds, runs and checks a unit under test. AI borrowed it for evaluation harnesses — the machinery that pipes a benchmark into a model and scores the output. Then agent builders borrowed it again for the loop, tools and context that wrap a model in production. Research papers more often call this last one scaffolding. 它不是从 AI 开始的。在软件工程里,test harness(测试挂载)是负责搭建、喂数据、运行并检查被测单元的那段代码。AI 先把它借去做评测 harness——把基准题送进模型、再给输出打分的那套机器。随后 agent 开发者又借了一次,用来指生产环境中包住模型的循环、工具与上下文。论文里更常把最后这一种称作 scaffolding(脚手架)。
Keep the three senses separate when you read, or you'll think two papers disagree when they're talking about different objects. 阅读时把这三层含义分清楚,否则你会以为两篇论文互相矛盾,其实它们说的根本不是同一个东西。
The parts list零件清单
Eight components. If your system is missing three of them, it's not an agent — it's a demo. 八个部件。如果你的系统少了其中三个,那它不是 agent,只是个 demo。
Context assembly上下文组装
≈ fuel injection≈ 燃油喷射Choosing, ordering, compressing and cutting what reaches the model this turn. Most of your quality wins live here, not in prompt wording.选择、排序、压缩、裁剪这一轮送进模型的内容。绝大多数质量提升来自这里,而不是提示词的措辞。
Tool interface工具接口
≈ transmission≈ 变速与传动Turning intent into action: schemas, argument validation, error messages the model can actually recover from. Designed for a model to use, not a human.把意图变成动作:接口定义、参数校验、以及模型真能自我纠正的错误信息。是为模型设计的界面,不是为人。
Control loop控制循环
≈ steering + throttle≈ 转向与油门Plan → act → observe → repeat, plus retries, branching and the stop condition. Fifty lines of code that decide everything.规划 → 执行 → 观察 → 重复,加上重试、分支和终止条件。五十行代码,决定一切。
Memory & state记忆与状态
≈ odometer + logbook≈ 里程表与行车记录Scratchpads, summaries of old turns, durable facts, working files. What survives when the window overflows.草稿区、历史摘要、长期事实、工作文件。当窗口溢出时,还能留下来的东西。
Guardrails & permissions护栏与权限
≈ brakes + limiter≈ 刹车与限速器Sandboxes, allow-lists, spend caps, human approval on irreversible actions. Never rely on the prompt to be the brake.沙箱、白名单、花费上限、不可逆操作的人工确认。永远不要把提示词当刹车用。
Observability可观测性
≈ dashboard + OBD port≈ 仪表盘与诊断口Full traces of every call, token, tool result and cost. Read ten real traces end to end; it beats any amount of theorising.完整记录每一次调用、token、工具结果和成本。把十条真实轨迹从头读到尾,胜过任何空想。
Evaluation suite评测集
≈ the test track≈ 试车场A frozen set of tasks with checkable outcomes. Without it you cannot tell a model upgrade from a harness regression.一组冻结的、结果可判定的任务。没有它,你分不清是模型升级了,还是 Harness 退化了。
Cost & latency governor成本与延迟调速
≈ gearbox + fuel gauge≈ 挡位与油量表Model routing, caching, parallelism, step budgets. Quality per dollar per second is the real spec sheet.模型路由、缓存、并行、步数预算。每一元、每一秒换到多少质量,才是真正的参数表。
Fifty lines, three tools, one scoreboard五十行代码,三个工具,一块记分板
Don't start with a framework. Start with a loop you can hold in your head, on a task where "correct" is machine-checkable — tests pass, the number matches, the file compiles. 别从框架开始。从一个你脑子里装得下的循环开始,选一个"对不对"能被机器判定的任务 —— 测试通过、数字对上、文件能编译。
# the whole idea, minus the yak-shaving
state = {goal, files, notes}
for step in range(MAX_STEPS): # stop condition = brakes
ctx = assemble(state) # fuel injection — most wins here
out = model(ctx, tools) # the ENGINE
if out.is_final: break
if violates(policy, out): halt() # guardrail, not a prompt
obs = run_tool(out.call) # the drivetrain
state = update(state, out, obs) # memory
log(step, ctx, out, obs, cost) # dashboard
score(result, rubric) # the test track
Pick one checkable task选一个可判定的任务
"Fix this failing test", "extract these 6 fields from this PDF". Vague tasks make ungradeable harnesses.比如"修好这个失败的测试""从这份 PDF 抽出这 6 个字段"。模糊任务只会造出无法打分的 Harness。
Write the loop by hand手写那个循环
No agent library for the first version. You need to feel every decision the framework would have hidden.第一版不要用 agent 库。你需要亲手感受框架会替你藏起来的每一个决定。
Define 3–7 tools定义 3 到 7 个工具
Read, run, write. Make the error messages instructive — a good tool teaches the model how to retry.读、执行、写。让报错信息本身具有指导性 —— 好工具会教模型怎么重试。
Freeze 20–50 eval cases冻结 20 到 50 个评测样例
From your real domain, with expected outcomes. Never edit them to make a run look good.来自你真实的业务领域,附带期望结果。永远不要为了跑分好看去改它们。
Ablate one part at a time一次只拆一个部件
Remove memory. Remove retries. Halve the context. Record the delta. This is the experiment that teaches you the most.去掉记忆。去掉重试。把上下文砍半。记录分差。这是收获最大的实验。
Then swap the engine最后换引擎
Same harness, two models. Compare that gap with your ablation gaps — now you know which half you're actually building.同一个 Harness,两个模型。把这个差距和拆件实验的差距对比 —— 你就知道自己真正在造的是哪一半。
Three caveats三个前提
The harness changes the horsepowerHarness 会改变马力
A torque curve belongs to the engine alone. A model's measured ability does not — it is a property of model × scaffold × task. Chain-of-thought prompting, self-consistency sampling and better agent interfaces all move benchmark scores with zero change to the weights. When someone quotes a model score, ask what harness produced it.扭矩曲线只属于引擎。模型被测出的能力却不是 —— 它是"模型 × 脚手架 × 任务"的属性。思维链提示、自洽采样、更好的 agent 接口,都能在权重完全不变的情况下改变基准分数。所以别人报一个模型分数时,先问:是哪套 Harness 跑出来的?
Wei et al., Chain-of-Thought Wang et al., Self-Consistency Yang et al., SWE-agent / ACI
A new engine can delete your vehicle新引擎会把你的车删掉
Cars don't become obsolete because the engine improved. Harnesses do. Every clever workaround you build to compensate for today's model weakness becomes dead weight when the next model ships — that's Sutton's bitter lesson wearing work clothes. So build thin where the model is improving fast (planning, reasoning, decomposition) and thick where it will never help you (permissions, data access, audit trails, cost control, your product's UI, your domain's ground truth).汽车不会因为引擎变强而报废,Harness 会。你为弥补当下模型缺陷而写的每一个巧妙补丁,等下一代模型发布就变成了负重 —— 这就是穿着工装的"苦涩教训"。所以:在模型进步最快的地方(规划、推理、任务分解)要薄;在模型永远帮不了你的地方(权限、数据访问、审计、成本控制、产品界面、行业事实)要厚。
The failures are semantic, not mechanical故障是语义的,不是机械的
A cracked piston fails in bounded, physical ways. A model fails by being confidently wrong, or by obeying an instruction hidden in a web page it just read. Prompt injection has no mechanical analogue — you can't torque-wrench your way out of an adversary who writes English. Brakes here mean sandboxes, least privilege and human approval on irreversible actions, not a stern system prompt.活塞裂了,故障是有界的、物理的。模型的故障是"自信地答错",或者听从了它刚读到的网页里藏着的一条指令。提示注入没有机械对应物 —— 面对一个会写自然语言的对手,扳手是拧不出安全的。这里的刹车是沙箱、最小权限、不可逆操作的人工确认,而不是一段语气严厉的系统提示。
And one loop the metaphor can't hold at all: vehicles don't build engines. But agent harnesses generate the traces, environments and reward signals used to train the next model. The vehicle is quietly reshaping the engine — which is why "just a wrapper" has been wrong for a while now. 还有一个比喻完全装不下的回路:车不会造引擎。但 agent 的 Harness 会产出用于训练下一代模型的轨迹、环境和奖励信号。车正在悄悄重塑引擎 —— 这也是"不过是个套壳"这句话早就不成立的原因。
Go read these six先读这六个
lm-evaluation-harness
EleutherAI's framework for scoring language models. The literal harness that produced many numbers you've quoted.EleutherAI 的语言模型评测框架。你引用过的很多分数,就是这套"挂载"跑出来的。
github.com/EleutherAI/lm-evaluation-harnessSWE-agent
Shows that designing the interface a model uses to browse and edit code — not the model — drives the score.证明了拉高分数的是模型浏览与修改代码所用的界面设计,而不是模型本身。
arxiv.org/abs/2405.15793ReAct
Thought → action → observation. The 2022 paper that most agent control loops are still a variation of.思考 → 行动 → 观察。2022 年那篇论文,如今多数 agent 控制循环仍是它的变体。
arxiv.org/abs/2210.03629SWE-bench
Real GitHub issues, graded by whether the tests pass. A test track where the harness is measurably half the car.真实的 GitHub issue,以测试是否通过来打分。在这个试车场上,Harness 实打实占了半台车。
swebench.comBuilding effective agents
Anthropic's argument for simple, composable workflows over elaborate frameworks. Read before you install anything.Anthropic 主张用简单可组合的工作流,而不是复杂框架。装任何库之前先读它。
anthropic.com/engineering/building-effective-agentsModel Context Protocol
An open standard for connecting tools and data to models — the attempt to standardise the engine mounts.把工具与数据接入模型的开放标准 —— 一次把"引擎支架"标准化的尝试。
modelcontextprotocol.ioA four-week training plan一份四周训练计划
Reading gives you the vocabulary. Only a measured ablation gives you the answer.读文献给你词汇,只有做过一次有数据的拆件实验,才能给你答案。
Run an eval harness yourself亲手跑一次评测 harness
Install lm-evaluation-harness, run one small open model on two tasks. Change the few-shot count and the prompt template. Watch the score move while the weights don't.装上 lm-evaluation-harness,用一个小型开源模型跑两个任务。改一下 few-shot 数量和提示模板。看着权重没变、分数却在动。
Write the fifty-line loop写出那五十行循环
Three tools: read file, run command in a sandbox, write file. A step limit and a spend cap. No framework, no abstractions you didn't type.三个工具:读文件、在沙箱里执行命令、写文件。加上步数上限和花费上限。不用框架,不用你没亲手敲过的抽象。
Thirty cases and an ablation table三十个样例与一张拆件表
Freeze 30 tasks from your own domain with checkable outcomes. Then remove memory, remove retries, halve the context, degrade the tool errors. Record every delta.从自己的领域冻结 30 个结果可判定的任务。然后去掉记忆、去掉重试、把上下文砍半、把工具报错变模糊。每一次的分差都记下来。
Swap engines, keep the vehicle换引擎,留下车
Same harness, two different models, same 30 cases. Then read one real agent codebase end to end — SWE-agent is a good first one — and mark which of your eight parts it has.同一个 Harness、两个不同模型、同样 30 个样例。然后把一个真实 agent 代码库从头读到尾(SWE-agent 是不错的第一个),标出它具备八个部件中的哪几个。
Your deliverable: one table. Rows = harness variants. Columns = models. Cells = score, cost, latency. If the spread across rows rivals the spread across columns, you've just proved the metaphor's most important half — and its limit — with your own data.你的交付物:一张表。行 = Harness 变体,列 = 模型,格子 = 分数、成本、延迟。如果行与行之间的差距能和列与列之间的差距相当,你就用自己的数据同时证明了这个比喻最重要的一半,以及它的边界。
Sources出处
Everything asserted above traces back to one of these. Primary sources first; read the papers, not the summaries of the papers.上面每一条判断都能追溯到下面某一项。优先读一手资料 —— 读论文本身,而不是论文的转述。
Origins & vocabulary词源与术语
- Test harness — WikipediaThe software-engineering original.软件工程中的原始定义。
- EleutherAI — lm-evaluation-harnessThe reference evaluation harness.最具代表性的评测 harness。
- Hugging Face — Open LLM LeaderboardA public scoreboard built on that harness.建立在该 harness 之上的公开榜单。
- METREvaluations research; uses "scaffolding" for agent harnesses.评测研究机构;用 "scaffolding" 指代 agent harness。
Loops, scaffolds & tools (papers)循环、脚手架与工具(论文)
- Chain-of-Thought Prompting — Wei et al., 2022Prompt structure changes measured capability.提示结构会改变被测能力。
- Self-Consistency — Wang et al., 2022Sampling strategy as part of the harness.采样策略也是 harness 的一部分。
- ReAct — Yao et al., 2022Reasoning interleaved with acting: the canonical loop.推理与行动交错:经典循环范式。
- Toolformer — Schick et al., 2023Models learning to call tools.让模型学会调用工具。
- Reflexion — Shinn et al., 2023Verbal self-feedback as a harness component.把语言化自反馈做成 harness 的部件。
- Tree of Thoughts — Yao et al., 2023Search over the model's outputs.在模型输出之上做搜索。
- Voyager — Wang et al., 2023Skill library and lifelong-learning loop.技能库与持续学习循环。
- Retrieval-Augmented Generation — Lewis et al., 2020Where context assembly began.上下文组装的起点。
Harnesses & benchmarksHarness 与基准
- SWE-bench — Jimenez et al., 2023Real repository issues, test-verified grading.真实仓库 issue,以测试验证打分。
- SWE-agent: Agent-Computer Interfaces — Yang et al., 2024The clearest evidence that interface design is capability.界面设计即能力,最清楚的证据。
- SWE-agent — project site & codeA readable agent codebase to study.一个适合精读的 agent 代码库。
- τ-bench — Yao et al., 2024Agents with tools, rules and real users.带工具、规则与真实用户的 agent 评测。
- Berkeley Function-Calling LeaderboardTool-use ability, measured.工具调用能力的量化测量。
- OpenAI EvalsFramework for writing your own eval set.用来写自己评测集的框架。
Engineering guides工程指南
- Anthropic — Building effective agentsWorkflows vs. agents; start simple.工作流与 agent 的区别;从简单开始。
- Anthropic — Effective context engineering for AI agentsThe fuel-injection chapter.对应"燃油喷射"那一章。
- OpenAI — A practical guide to building agents (PDF)Loop, tools, guardrails, handoffs.循环、工具、护栏、交接。
- Model Context ProtocolStandardising tool and data connections.统一工具与数据的接入方式。
- DSPy — Stanford NLPProgramming, and optimising, the scaffold itself.把脚手架当程序来写,并自动优化它。
- Rich Sutton — The Bitter LessonWhy harnesses get deleted.Harness 为什么会被删掉。
Risk, guardrails & brakes风险、护栏与刹车
- OWASP Top 10 for LLM ApplicationsThe failure taxonomy to design against.设计防护时要对照的故障分类。
- Simon Willison — Prompt injection seriesYears of worked examples; still unsolved.多年的实例记录;至今仍未解决。
- NVIDIA NeMo GuardrailsOne concrete implementation of policy brakes.策略型刹车的一种具体实现。
- NIST AI Risk Management FrameworkGovernance vocabulary for the whole vehicle.面向"整车"的治理术语体系。