The Ablation Table

A

The four weeks, step by step四周,逐步操作

01

Week 1 · Reproduce第一周 · 复现

Run an eval harness yourself亲手跑一次评测框架

The point is not the score. The point is to watch a number move a long way while the model weights never change at all.重点不是分数,而是亲眼看到:权重一点没动,分数却大幅移动。

Create a clean virtual environment. Install lm-evaluation-harness and pin the exact commit in a notes file.建一个干净的虚拟环境,安装 lm-evaluation-harness,并把使用的 commit 号写进笔记。
Pick one small open model that fits your laptop or one cheap GPU hour. Small is the feature, not the compromise.选一个能跑在你笔记本上、或一小时廉价 GPU 上的小型开源模型。“小”是优点,不是妥协。
Run two tasks — one knowledge, one reasoning. Keep the raw JSON output; never keep only the headline number.跑两个任务 —— 一个偏知识、一个偏推理。保存原始 JSON,不要只留下那个总分。
Change only the few-shot count: 0 → 5. Re-run. Write the delta down by hand.只改 few-shot 数量:0 → 5。重跑,手写记录差值。
Change only the prompt template. Re-run. Write that delta down too.只改 prompt 模板。重跑,同样记录差值。
Write one paragraph for yourself: how much of the “model quality” you read about is actually harness.给自己写一段话:你平时读到的“模型能力”,有多少其实来自评测框架本身。

02

Week 2 · Build第二周 · 自建

Write the fifty-line loop写出那五十行的循环

One file, no framework, no abstraction you did not type. An agent is a while loop with tools, a budget and a stop condition.一个文件,没有框架,没有一行不是你自己敲的抽象。所谓 agent,就是带工具、带预算、带停止条件的 while 循环。

A message list, a model call, a while loop. That is the whole engine.一个消息列表、一次模型调用、一个 while 循环 —— 引擎就这些。
Exactly three tools: read a file, run a command in a sandboxed container, write a file.只给三个工具:读文件、在沙箱容器里执行命令、写文件。
A hard step limit (say twelve) and a spend cap that raises an exception, not a warning.设硬性步数上限(比如 12 步),以及一个超支就抛异常的花费上限 —— 不是警告,是异常。
Append every tool call and every result to a JSONL file. This log is your telemetry for week 3.把每次工具调用与返回都追加进 JSONL 文件 —— 这就是第三周要用的遥测数据。
Give it one real chore from your own week. Watch precisely where and how it fails.丢给它一件你本周真实的杂活,盯住它到底在哪一步、以什么方式失败。
Resist adding a fourth tool for seven days. Constraint is what makes week 3 readable.忍住七天不加第四个工具。正是这个约束,让第三周的结果可读。

03

Week 3 · Measure第三周 · 测量

Thirty cases and an ablation table三十个用例,一张消融表

Thirty is enough to see a real effect and small enough that you will actually freeze the set instead of endlessly editing it.三十个,足以看出真实差异,又少到你真的会把它冻结下来,而不是反复修改。

Freeze 30 tasks from your own domain, each with a checkable pass/fail outcome.从你自己的业务里冻结 30 个任务,每个都要有可判定的通过/失败标准。
Write the checker before you write the prompt. If you cannot check it, it is not a case.先写判分器,再写 prompt。判不了的,就不算用例。
Baseline run: score, cost, wall-clock latency. Three seeds, so you know your noise floor.跑基线:分数、成本、耗时。跑三个随机种子,先摸清噪声底线。
Ablate one part per run: remove memory; remove retries; halve the context; degrade the tool error messages.每次只消融一个部件:去掉记忆;去掉重试;上下文减半;把工具报错信息劣化。
One row per variant. Never change two things in one run — that row becomes unreadable forever.每个变体一行。绝不在一次运行里改两件事 —— 那一行会永远失去意义。
Record every delta, including the ones that move the wrong way. Those are the honest rows.记录每一个差值,包括方向“不对”的那些 —— 那才是诚实的行。

04

Week 4 · Compare第四周 · 对比

Swap engines, keep the vehicle换发动机,不换车

Same harness, same thirty cases, a second model. Then read one real agent codebase end to end and see what a serious one keeps.同一套框架、同样三十个用例,换第二个模型。然后完整读一份真实的 agent 代码,看看成熟实现留下了什么。

Run model B through the identical harness. Change nothing else — not the prompt, not the retries.用完全相同的框架跑模型 B。其他一律不动 —— prompt 不动,重试不动。
Fill the grid: rows = harness variants, columns = models, cells = score, cost, latency.填格子:行 = 框架变体,列 = 模型,格 = 分数、成本、延迟。
Read SWE-agent end to end — it is a good first real codebase.从头到尾读一遍 SWE-agent —— 作为第一份真实代码库很合适。
Mark which of your eight parts it has, which it does without, and where it spends its complexity.标出你那八个部件里它有哪些、省掉了哪些、复杂度花在了哪里。
Write the limit of the “engine and vehicle” metaphor in three sentences, backed by your own rows.用三句话写出“发动机与整车”这个比喻的边界,并用你自己的数据行支撑它。

01Model模型

02Prompt / policy提示词 / 策略

03Tools工具

04Memory记忆

05Loop control循环控制

06Error handling错误处理

07Budget limits预算上限

08Eval & telemetry评测与遥测

B

The deliverable最终交付物

Print it. Fill it by hand if you like.打印出来,手写填也可以。

One table.就一张表。

Harness variant框架变体	Model A模型 A	Model B模型 B	Δ across models跨模型差
Baseline (full loop)基线(完整循环)	—score—cost—p50 s	—score—cost—p50 s	—pts
− memory− 记忆	—score—cost—p50 s	—score—cost—p50 s	—pts
− retries− 重试	—score—cost—p50 s	—score—cost—p50 s	—pts
½ context上下文减半	—score—cost—p50 s	—score—cost—p50 s	—pts
Degraded tool errors劣化的工具报错	—score—cost—p50 s	—score—cost—p50 s	—pts
Spread down the rows行内跨度	—pts	—pts	compare the two spreads比较两个跨度

If the spread down the rows rivals the spread across the columns, you have proved the metaphor's most important half — the vehicle matters as much as the engine — and found its limit, with your own data rather than someone else's slide. 如果行方向的跨度能与列方向的跨度相当,你就用自己的数据(而不是别人的 PPT)证明了这个比喻最重要的一半 —— 整车和发动机同等重要 —— 也同时找到了它的边界。

Rows = variants · Columns = models · Cells = score, cost, latency行 = 变体 · 列 = 模型 · 格 = 分数、成本、延迟

C

The same table, in sustainability operations同一张表,用在可持续运营里

A cross-field role — sustainability across global supply chain and product, for AI datacenter and AI PC — is exactly where this method pays. You are already asked to compare things that are not comparable: a cloud inference workload against an on-device one, a supplier's estimate against a measured value. The ablation table is the honest way to do it.当你的角色横跨全球供应链与产品的可持续性,又同时面对 AI 数据中心与 AI PC,这套方法正好适用。你本来就被要求去比较“不可比”的东西:云端推理与端侧推理、供应商估算值与实测值。消融表就是诚实地做这件事的方式。

Change one thing per run, and the number you report finally belongs to you. 每次只改一件事,你报出去的那个数字,才真正属于你。

Choose 30 real work items: supplier questionnaire triage, product carbon footprint data extraction, compliance spec parsing, BOM reconciliation.选 30 项真实工作:供应商问卷分类、产品碳足迹数据提取、合规规范解析、BOM 对账。
Define “pass” as: a human reviewer would ship it unchanged. Nothing softer.把“通过”定义为:人工复核后可以原样交付。不接受更宽松的定义。
Instrument energy from the start — device or node power × wall-clock, or the provider's reported kWh. Log location-based and market-based carbon separately.从第一天就接入能耗测量 —— 设备/节点功率 × 运行时长,或云厂商报出的 kWh。基于位置与基于市场的碳排分别记录。
Ablate the operating choices, not only the model: quantization, context length, retrieval on/off, retries, batch size, cloud GPU vs AI PC NPU, scheduling to low-carbon hours.消融的是运营选择,而不只是模型:量化、上下文长度、检索开关、重试、批大小、云端 GPU 对比 AI PC 上的 NPU、以及调度到低碳时段。
Report per useful outcome — grams of CO₂e per accepted answer — never per token. Tokens are not a unit of value.按“有效产出”报告 —— 每个被采纳答案的 gCO₂e —— 而不是按 token。token 不是价值单位。
Carry the winning row into your Scope 3 and product footprint story with a written methodology note: boundary, factors, date, and what you did not measure.把胜出的那一行写进 Scope 3 与产品碳足迹叙述,并附上方法学说明:边界、排放因子、日期,以及你没有测量的部分。

Operating variant运营变体	Datacenter GPU数据中心 GPU	AI PC (NPU, on-device)AI PC(端侧 NPU)
Baseline, FP16基线,FP16	—pass %—kWh/1k—gCO₂e	—pass %—kWh/1k—gCO₂e
INT8 quantizedINT8 量化	—pass %—kWh/1k—gCO₂e	—pass %—kWh/1k—gCO₂e
− retrieval− 检索	—pass %—kWh/1k—gCO₂e	—pass %—kWh/1k—gCO₂e
− retries− 重试	—pass %—kWh/1k—gCO₂e	—pass %—kWh/1k—gCO₂e
Scheduled, low-carbon hours调度至低碳时段	—pass %—kWh/1k—gCO₂e	—pass %—kWh/1k—gCO₂e

Add embodied carbon as a separate line, not inside the cell — an AI PC row hides a device that was manufactured once. 把隐含碳单列一行,不要塞进格子里 —— AI PC 那一行背后,是一台只制造过一次的设备。

D

References for further learning延伸学习参考

The four weeks, step by step四周,逐步操作

Run an eval harness yourself亲手跑一次评测框架

Write the fifty-line loop写出那五十行的循环

Thirty cases and an ablation table三十个用例,一张消融表

Swap engines, keep the vehicle换发动机,不换车

The deliverable最终交付物

One table.就一张表。

The same table, in sustainability operations同一张表,用在可持续运营里

References for further learning延伸学习参考

Harnesses & measurement评测框架与测量

Agent codebases & designAgent 代码与设计

Energy & carbon of compute算力的能耗与碳排

Supply chain, product & disclosure供应链、产品与披露