我给 coding agent 造了一个记忆能力包：它现在可以通过标准 MCP 工具语义检索我的跨项目知识库，并且有一个结构上不可能把记忆写坏的写入口。

真正决定成败的不是某个模型或某段代码，而是三条原则：

1. 把不可逆的操作从工具面里拿掉。删除、原地改写这类动作根本不暴露给 agent，安全性就成了结构约束，而不是提示词约束。
2. 把真相源和派生索引分开。真相是 Markdown（可 diff、可回退），向量索引只是可随时丢弃的加速结构。索引坏了，重建即可。
3. 把每个结论都变成可复现的锚点。写代码的是 AI，那就更不能用”看起来对”当验收标准。

最终交付：单测 163 通过、写路径确定性套件 25/25、跨会话盲测 recall 通过、索引自洽（135 条 = 135 个向量点）。

一、问题：agent 的”记忆”缺什么

我有一个跨项目使用的个人 Markdown 知识库，加上三个项目仓库。问题是：coding agent 只能靠 grep 去翻它。这带来两个真实的痛。

检索是字面的。 我记的是”决策质量与竞争性假设（ACH）”，问的是”怎么避免拍脑袋定方案”——grep 命中不了。agent 于是把已经记录过的结论重新推导一遍，浪费上下文，还可能推歪。

写入是不安全的。 让 agent 直接往 Markdown 里写，它可能写成重复条目、把一条记忆拆成五条、原地覆盖旧结论，甚至删文件。知识库最怕的不是搜不到，是悄悄坏掉。

所以目标不是”再做一个 RAG”，而是：让 agent 能可靠地回忆，并且有一个值得信任的写入口。

二、形状：从”一个 RAG 应用”变成”一个能力包”

原来的项目是一个政策法规问答的 RAG web。升级后重排成三个模块：

1
2
3

ragcore/       可复用核心（检索、向量库、文档处理，零 Web 依赖）
legal_web/     适配层实例 / 回归锚点（原来的 RAG web）
memory_agent/  hero：记忆能力包（MCP + skill）

把 hero 切成记忆能力包之后，原来的 web 退成”适配层 + 回归锚点”：它仍然必须能启动、能评测，用来证明核心没被改坏。这是很划算的一手——你永远有一个跑得起来的旧东西当对照。

三、五条核心决策

3.1 Markdown 是真相源，向量索引是派生物

1
2
3

可写记忆 = 全局知识库的 Markdown 条目（frontmatter: id/type/tags/status/supersedes…）
只读语料 = 项目仓库的 Markdown 文档（writable:false）
向量索引 = 从上面两者重建出来的、可随时丢弃的加速结构

记忆的语义是”可版本化、可回退、可人读”。数据库或向量库做主存，都做不到 git log 一眼看变更、git revert 一键回退。这个决定后面反复救了场：出过”索引 0 个点、但 manifest 还写着 60 条”的静默故障，因为真相源没丢，重建就完事。

3.2 写入网关：不覆盖、不删除、不原地编辑

对外只暴露 MCP 工具，不暴露任何裸文件写。写入必须过一次网关：

1

search → dedup → validate(frontmatter) → write → git commit → incremental refresh

生命周期语义只有两种，都不是”删除”：

两者都是破坏性操作，默认 confirm=false 只返回预览，用户明确同意后才落盘。

还有一条工程细节：每次写入 = 只包含本次触及文件的一个 git commit。多会话并发时很关键——如果 git add -A，就会把别的会话正在编辑的脏文件一起提交进去。

1
2
3
4
5
6
7
8
9
10
11
12

graph LR
  A[agent] -->|memory_add| G[写入网关]
  A -->|memory_supersede / archive| G
  G -->|1 搜索去重| I[(派生索引)]
  G -->|2 校验 frontmatter| V[KB 规范]
  G -->|3 落盘 + 单文件 commit| M[Markdown 真相源]
  G -->|4 增量刷新| I
  M -.重建.-> I
  style A fill:#e8f0fe,stroke:#4285f4
  style G fill:#fff4e5,stroke:#f4a742
  style M fill:#e6f4ea,stroke:#34a853
  style I fill:#fce8e6,stroke:#ea4335

3.3 索引一致性：代目录 + 指针原子切换

重建索引是长活（CPU 上约 6 分钟 / 60 条），中途会被超时、kill 或崩溃打断。朴素做法（清空集合再重建）如果中途死掉，就留下”空索引 + 陈旧 manifest“——get 照常、search 返回空，不核对根本发现不了。这个坑真实发生过。

现在的做法：

• 每次全量重建写在新代目录 vector_db/gen-N/；
• 全部建好并通过自洽核对（manifest 条数 == 集合点数）后，原子切换 CURRENT 指针；
• 中断只留下一个没被接管的代目录，旧代继续服务；
• 检索前核对自洽性，不一致就显式报错，绝不静默返回空。

3.4 一个常驻 daemon + 每会话一个瘦代理

这是整个项目里最”物理”的决策，因为它由一次 OOM 逼出来。

最初每个 opencode 会话各拉起一份 stdio MCP，各加载一份 BGE-M3 嵌入模型。BGE-M3 权重 2.17GB + torch 运行时 + 加载峰值 ≈ 每进程 3.9GB 私有内存。三条并行会话 = 约 12GB，直接把系统 commit 打满，结果是 Out of memory、卡死、uv_spawn 失败。

1
2
3
4
5
6
7
8
9
10
11

graph TD
  S1[会话 1] --> P1[瘦代理]
  S2[会话 2] --> P2[瘦代理]
  S3[会话 3] --> P3[瘦代理]
  P1 --> D[常驻 MCP daemon<br/>唯一一份 BGE-M3]
  P2 --> D
  P3 --> D
  style D fill:#e6f4ea,stroke:#34a853
  style P1 fill:#e8f0fe,stroke:#4285f4
  style P2 fill:#e8f0fe,stroke:#4285f4
  style P3 fill:#e8f0fe,stroke:#4285f4

N 个会话的内存从 N × 3.9GB 降到 1 × 3.9GB + N × 几十 MB。配套还解决了三件事：

• 代理必须幂等拉起 daemon。N 个会话同时冷启动会各自 spawn 一个 daemon（又回到 N 份模型），所以用启动权文件锁：只有一个进程负责拉起，其余只等健康检查。
• 同一进程内 Qdrant local mode 也不能并发开 client，于是用一个进程内锁把并发串行化。
• 默认不预热模型：启动私有内存从 3953MB 降到 54MB。

3.5 只读语料：带标签的多仓库文档

最后把三个项目仓库的文档也纳入检索。几个取舍：

• 只索引 .md，不索引代码。代码对”回忆决策”没有价值，进语料只会稀释检索。
• 仓库清单不硬编码。三个仓库是机器本地绝对路径，写进可发布代码就是”换机器即坏 + 泄露目录结构”。改成 gitignored 的 JSON 清单 + 一份 example 模板，环境变量仍可覆盖。
• 加 <label>/ 前缀消歧义。三个仓库都有 README.md、CONTEXT.md，裸相对路径会让只读条目 id 互撞，去重时静默丢条。现在来源是 agent-infra/docs/adr/0007-....md 这种带仓库名前缀的形式。
• 排除噪声：生成物、工具缓存一律不进——其中一个仓库光 outputs/ 就有 1339 个 markdown 抽取产物，全进索引就是灾难。

改配置有个必须记住的约束：配置在 import 时求值，改完必须重启 daemon。否则 daemon 拿着旧的只读根做增量刷新，会把”不在我语料里”的条目当孤儿删掉。

四、六个可复用的坑

1. stdio MCP：stdout 就是协议通道。 底层库的 logger 默认把日志写到 stdout，一行就能腐蚀 JSON-RPC。必须在 import 它之前把 root logger 抢配到 stderr。
2. 模块名会遮蔽包。 把 config.py 放在了不该放的位置，结果在脚本模式下遮蔽了另一个顶层 config 包。改名 + 统一前缀绝对导入。
3. Qdrant local mode 的两个反直觉语义。 (a) 独占锁在 client 构造期持有、close() 释放，所以不要长持 client，按操作开/关（实测约 19ms/次）；(b) 删集合后再用同名建集合，磁盘上的旧点会复活——清空集合要用空 filter 删光点。
4. “命令返回了” ≠ 成功。 重建进程被 kill，命令看起来正常结束，但状态是坏的。长活必须后台启动 + 输出重定向 + 独立轮询进度 + 完成后自洽核对。
5. 客户端超时 ≠ 写入失败。 MCP 客户端 20s 超时，但服务端可能已经 commit 成功。超时后要核对真相源，不能直接重试（重试会命中去重，只返回 duplicate）。
6. 去重阈值要校准，不要拍脑袋。 最初拍 0.92，在真实知识库上量了一下：不同条目的最近邻相似度 ≤0.792，精确重加最低 0.949，于是改成 0.88 落在间隔中部。

五、证据，而不是感觉

“AI 写的代码”最大的风险是看起来很对。所以整个过程尽量把结论变成机器可验的锚点：

几个刻意的测试口径：

• 只断言外部行为。sandbox 只检查工具返回值、落盘文件、git 状态、规范校验输出，不碰内部向量库调用。这样重构内部实现不会把测试改成”测实现”。
• 真实知识库只读。sandbox 克隆已提交态到临时目录，跑完用 git status 证明真实知识库未变。
• 锚点纪律。结构怎么重构，基准锚点（单测 + 导入冒烟 + 启动冒烟）必须每天可复现。

六、方法论：让 AI 开发可验证

这个项目的执行者是 AI（我提供思路和裁决，AI 写代码）。跑下来最有价值的不是某段代码，而是几条把”AI 生产力”变成”可验证结果”的纪律：

• 请求先进路由：缺陷走根因流程、新功能走决策闸门、实验只进 experiments/<name>/（先写记录再跑，无结论不算完成）。
• 一个单元 = 一个 commit：能一句话说清 + 能独立验证的增量就提交，不攒大提交。
• 硬决策必须留 ADR：难逆转、反直觉、真权衡的，写进 docs/adr/。这个项目留了 14 篇，覆盖每一次”为什么这么选”。
• 同一目标失败三次就停手调研，不要原地重试。
• 收工过健康闸门：git 干净、文档与目录树一致、无垃圾文件、实验有结论、决策有落点。

总结

这个 MVP 真正的产出不是”一个能搜的记忆库”，而是三件可复用的东西：

1. 结构约束 > 提示词约束。把删除、原地改写从工具面拿掉，agent 想做坏事也做不到。
2. 真相源与派生索引分离。Markdown 负责正确与可回退，索引负责快；前者坏了不可接受，后者坏了重建即可。
3. 证据文化。单测、确定性 sandbox、盲测 recall、自洽核对——每一个”完成”都要有机器可验的锚点。

下一步是把”检索有多好”从感觉变成可比数字（BEIR 子集 nDCG@10），以及向量 / 关键词 / 锚点的混合检索融合优化。但那是另一个故事了。

I built a memory capability package for coding agents: it can now semantically search my cross-project knowledge base through standard MCP tools, and it has a write path that is structurally incapable of corrupting memory.

What actually decided success wasn’t a model or a particular piece of code. It was three principles:

1. Remove irreversible operations from the tool surface. Delete and in-place rewrite are never exposed to the agent, so safety becomes a structural constraint instead of a prompt-level one.
2. Separate the source of truth from the derived index. The truth is Markdown (diffable, revertible); the vector index is a disposable accelerator. If the index breaks, rebuild it.
3. Turn every conclusion into a reproducible anchor. When an AI writes the code, “looks right” is even less acceptable as an acceptance criterion.

The final deliverable: 163 unit tests passing, a write-path determinism suite at 25/25, a cross-session blind recall test passing, and a self-consistent index (135 entries = 135 vector points).

1. The problem: what an agent’s “memory” is missing

I keep a personal Markdown knowledge base used across projects, plus three project repos. The problem: a coding agent can only grep through it. Two real pains follow.

Retrieval is literal. I stored “decision quality and Analysis of Competing Hypotheses (ACH),” then asked “how do I avoid making calls from the gut” — grep finds nothing. So the agent re-derives conclusions it already recorded, burning context and sometimes getting them wrong.

Writes are unsafe. Let an agent write Markdown directly and it might create duplicates, split one memory into five, overwrite an old conclusion in place, or just delete a file. The worst outcome for a knowledge base isn’t “can’t find it” — it’s quietly rotting.

So the goal wasn’t “another RAG.” It was: reliable recall, plus a write entry point you can trust.

2. Shape: from “a RAG app” to “a capability package”

The original project was a policy-and-law Q&A RAG web app. After the upgrade it was reorganized into three modules:

1
2
3

ragcore/       reusable core (retrieval, vector store, document handling; zero web deps)
legal_web/     adapter instance / regression anchor (the original RAG web app)
memory_agent/  the hero: the memory capability package (MCP + skill)

Once the hero became a memory package, the old web app was demoted to “adapter + regression anchor”: it must still boot and still be evaluable, proving the core wasn’t broken. That’s a great deal — you always have a working old thing to compare against.

3. Five core decisions

3.1 Markdown is the source of truth; the vector index is derived

1
2
3

writable memory   = Markdown entries in the global KB (frontmatter: id/type/tags/status/supersedes…)
read-only corpus  = Markdown docs from project repos (writable:false)
vector index      = a rebuildable, disposable accelerator over both

The semantics of memory are “versionable, revertible, human-readable.” A database or vector store as primary storage can’t give you git log for at-a-glance changes or git revert for one-command rollback. That decision saved the day repeatedly later: we hit a silent failure where “the index has 0 points but the manifest still lists 60 entries.” Because the source of truth was intact, a rebuild was all it took.

3.2 The write gateway: no overwrite, no delete, no in-place edits

Only MCP tools are exposed; no raw file write, ever. Every write goes through a gateway:

1

search → dedup → validate(frontmatter) → write → git commit → incremental refresh

There are exactly two lifecycle semantics, and neither is “delete”:

Both are destructive, so by default confirm=false returns a preview only; nothing is written until the user explicitly agrees.

One more engineering detail: every write is a git commit that contains only the files it touched. Under concurrent sessions that matters — git add -A would sweep in some other session’s work-in-progress.

1
2
3
4
5
6
7
8
9
10
11
12

graph LR
  A[agent] -->|memory_add| G[Write Gateway]
  A -->|memory_supersede / archive| G
  G -->|1 search & dedup| I[(Derived Index)]
  G -->|2 validate frontmatter| V[KB Rules]
  G -->|3 write + single-file commit| M[Markdown Source of Truth]
  G -->|4 incremental refresh| I
  M -.rebuild.-> I
  style A fill:#e8f0fe,stroke:#4285f4
  style G fill:#fff4e5,stroke:#f4a742
  style M fill:#e6f4ea,stroke:#34a853
  style I fill:#fce8e6,stroke:#ea4335

3.3 Index consistency: generations plus an atomic pointer swap

A full rebuild is a long job (~6 minutes per 60 entries on CPU) and can be interrupted by a timeout, a kill, or a crash. The naive approach — clear the collection, then rebuild — leaves “empty index + stale manifest“ if it dies midway: get works, search returns nothing, and you won’t notice unless you check. We hit exactly that.

The current approach:

• Each full rebuild is written into a new generation directory vector_db/gen-N/;
• Only after it is complete and passes a consistency check (manifest count == point count) do we atomically swap the CURRENT pointer;
• An interruption leaves an unadopted generation directory; the old one keeps serving;
• Before serving a search we verify consistency and raise loudly if it fails — never a silent empty result.

3.4 One resident daemon plus a thin per-session proxy

This is the most “physical” decision in the project, and it was forced by an OOM.

Initially each opencode session spawned its own stdio MCP server, each loading its own BGE-M3 embedding model. The weights are 2.17GB, plus the torch runtime and load peak ≈ 3.9GB private memory per process. Three parallel sessions ≈ 12GB, which exhausted system commit and produced Out of memory, hangs, and uv_spawn failures.

1
2
3
4
5
6
7
8
9
10
11

graph TD
  S1[Session 1] --> P1[Thin Proxy]
  S2[Session 2] --> P2[Thin Proxy]
  S3[Session 3] --> P3[Thin Proxy]
  P1 --> D[Resident MCP daemon<br/>single BGE-M3]
  P2 --> D
  P3 --> D
  style D fill:#e6f4ea,stroke:#34a853
  style P1 fill:#e8f0fe,stroke:#4285f4
  style P2 fill:#e8f0fe,stroke:#4285f4
  style P3 fill:#e8f0fe,stroke:#4285f4

N sessions went from N × 3.9GB to 1 × 3.9GB + N × tens of MB. Three more things came with it:

• The proxy must start the daemon idempotently. N sessions cold-starting simultaneously would each spawn a daemon (back to N models), so a startup-ownership file lock ensures only one process spawns while the rest just wait for health checks.
• Within one process, Qdrant local mode can’t open a second client concurrently either, so a process-level lock serializes access.
• Warmup is off by default: startup private memory dropped from 3953MB to 54MB.

3.5 Read-only corpus: labeled documents across multiple repos

Finally, I brought three project repos’ documents into retrieval. A few trade-offs:

• Index .md only, never code. Code is worthless for “recalling decisions” and only dilutes retrieval.
• Don’t hardcode the repo list. The three repos are machine-local absolute paths; baking them into publishable code means “breaks on another machine, and leaks directory structure.” Instead: a gitignored JSON list plus a committed example template, with an env override still available.
• Prefix sources with <label>/ to disambiguate. All three repos have README.md and CONTEXT.md; bare relative paths make read-only entry ids collide, and de-duplication silently drops entries. Now sources look like agent-infra/docs/adr/0007-....md.
• Exclude noise: generated artifacts and tool caches never enter. One repo alone had 1339 markdown extraction outputs under outputs/; indexing all of them would be a disaster.

One constraint you must remember when changing config: it is evaluated at import time, so restart the daemon after editing it. Otherwise the daemon refreshes incrementally using stale read-only roots and deletes entries it thinks are orphans.

4. Six reusable scars

1. stdio MCP: stdout is the protocol channel. The underlying library’s logger writes to stdout by default; one line corrupts JSON-RPC. You must grab the root logger and point it at stderr before importing it.
2. Module names can shadow packages. A config.py placed in the wrong spot shadowed another top-level config package in script mode. Renamed it and switched to consistently prefixed absolute imports.
3. Two counterintuitive semantics of Qdrant local mode. (a) The exclusive lock is held at client construction and released by close(), so don’t hold a client long-term — open/close per operation (~19ms each, measured). (b) Dropping a collection and recreating it with the same name resurrects the old on-disk points — clear by deleting all points with an empty filter.
4. “The command returned” ≠ success. Kill the rebuild process and the command looks like it finished, but state is broken. Long jobs need background start + redirected output + independent progress polling + a consistency check at the end.
5. Client timeout ≠ write failure. An MCP client may time out at 20s while the server has already committed. After a timeout, check the source of truth — don’t retry (a retry hits dedup and just returns duplicate).
6. Calibrate the dedup threshold; don’t guess it. The first guess was 0.92. Measured on the real KB: nearest neighbors of distinct entries were ≤0.792, and exact re-adds started at 0.949 — so 0.88 sits in the middle of the gap.

5. Evidence, not vibes

The biggest risk with “AI-written code” is that it looks right. So we tried to turn every conclusion into a machine-verifiable anchor:

A few deliberate testing stances:

• Assert external behavior only. The sandbox checks tool responses, files on disk, git state, and spec-validation output — never internal vector-store calls. That way refactoring internals can’t turn tests into tests-of-implementation.
• Keep the real KB read-only. The sandbox clones the committed state into a temp dir and proves with git status that the real KB is untouched.
• Anchor discipline. However the structure is refactored, the baseline anchors (unit tests + import smoke + boot smoke) must be reproducible every day.

6. Methodology: making AI development verifiable

The executor on this project was an AI (I supplied direction and made the calls; the AI wrote the code). The most valuable output wasn’t any particular chunk of code — it was a set of disciplines that turn “AI productivity” into “verifiable results”:

• Route requests first: defects go through a root-cause loop, features through a decision gate, experiments into experiments/<name>/ (write the record before running; no conclusion means not done).
• One unit = one commit: commit any increment you can explain in one sentence and verify independently; don’t hoard a giant commit.
• Hard decisions get an ADR: anything hard to reverse, counterintuitive, or a genuine trade-off goes into docs/adr/. This project has 14, covering every “why this choice.”
• If the same goal fails three times, stop and research — don’t retry in place.
• Run a health gate at wrap-up: clean git, docs matching the real tree, no junk files, experiments with conclusions, decisions with a home.

Summary

The real output of this MVP isn’t “a searchable memory store.” It’s three reusable things:

1. Structural constraints beat prompt constraints. Take delete and in-place rewrite off the tool surface, and the agent can’t do harm.
2. Separate the source of truth from the derived index. Markdown owns correctness and revertibility; the index owns speed. The former must never break; the latter can always be rebuilt.
3. A culture of evidence. Unit tests, a deterministic sandbox, blind recall, consistency checks — every “done” needs a machine-verifiable anchor.

Next up: turning “how good is retrieval” from a feeling into a comparable number (BEIR subset nDCG@10), and optimizing the fusion of vector / keyword / anchor signals. But that’s another story.

项目做完，最值钱的不是结果，是踩过的坑和摸清的流程。但沉淀的关键从来不是”多写几条”，而是先分流：

• 流程/SOP（”怎么做”）→ Agent Skill（可执行的步骤）；
• 事实/数据（”是什么”）→ 知识库；
• 短而硬的规则（”见到即改”）→ AGENTS。

再加一条同样重要的原则：不要从零造 skill。先找现成的参考再裁剪，比自研稳。

一、怎么决定放哪：先问”怎么做”还是”是什么”

这是全套沉淀的第一颗扣子。问错，代价是双向的：

• 把事实塞进 skill → skill 越来越长，最后没人读；
• 把方法塞进知识库 → 它不会被触发，等于没沉淀。

我们的全局 AGENTS.md 就是从”堆规则”演化来的——现在只剩 Tripwires（绊线）+ Gates（闸门） 两层，流程全部下沉到 skill。

二、知识库上的两个决策

知识库这一块，我们做了两个明确的决策（各留了一份决策记录）：

决策一：纯 Markdown + skill，先不上 RAG。

• 用 INDEX.md + 文本检索定位，正文是原子化条目；
• 与人的 Obsidian 笔记分开——那是给人看的，没有 agent 检索路径；agent 知识库要的是机器友好的结构；
• 验收标准是”新开一个会话，agent 只靠 skill + 索引就能找到并引用“；
• RAG/向量检索明确留到第二期，不改变 Markdown 文件本身。

理由很直接：没有服务就没有第二个会漂移的产物，离线、可 diff、可携带。

决策二：”有界索引、无界正文” + 检索阶梯。

• 根索引只列域（有上限），每个域下面才是条目；域索引由脚本生成；
• 条目有生命周期（current / draft / superseded / archived）与去重/覆盖规则——写之前先搜，相近的就合并或取代，绝不追加重复；
• 检索能力按触发条件分阶升级（L0 索引 → L1 文本检索 → L2 本地 FTS5 → L3 向量），没有明确信号就不上。

核心认识：增长与召回是两个问题——结构限制增长，向量只改善召回。用一个不断吞入的向量库，并不能解决”索引无限膨胀”。

三、四个 skill 各自装了什么

落下来的流程型 skill 就四个，都不长：

1. research-first —— 什么时候该停下来调研

• 触发：新栈（平台/框架/硬件/库）、框架 vs 手搓的选型、同一目标失败 3 次；
• 计数单位是目标不是错误——同一个目标换不同报错，失败数照累加；
• 一手源优先（官方文档、版本矩阵、issue），记录确切版本 + URL；先查知识库；
• 必产一份调研备忘录（假设 / 证据 / 版本矩阵 / 否决的选项 / 唯一下一步 / 置信度）。

2. expensive-run-gates —— 花钱前怎么设闸

• 闸 1 干跑：产出配置卡，和参考实现（anchor）逐项 diff——注意”自洽 ≠ 可比”，同门兄弟之间 diff 通过只证明自洽；
• 闸 2 最小对照 + 机器可验证据：format_check（格式对）、mask_check（标签不是全 -100）、run_config；两个截然不同的包装拿到一模一样的分数，就要怀疑它们共享同一个缺陷（我们的掩码 bug 就是这样暴露的）；
• 闸 3 人放行：给出预期成本、墙钟、成功信号、中止信号，人签字才花钱；
• 安全网：checkpoint（见下）；
• 还规定了一条证据契约：每次昂贵运行都往 run 目录写 run_config / format_check / mask_check / receipt，结论只从这些工件里取，绝不从控制台一行字。

3. checkpoint-design —— 在哪停、谁验证、存什么

• 本质：checkpoint 是在问题复利之前抓住它——两件事：验证（抓住它）+ 落盘（不必重做）；
• 在哪停：列出这次运行的承重假设，按”错了多贵 × 多难撤销 × 下游依赖多重”排序，在最高的那处停；不能改变决策的检查，跳过；
• 谁验证：机械/可查/可逆 → AI + 能定论的最小实验；品味/权衡/不可逆 → 人；重复、高风险、易错 → 独立或对抗式检查；有争议 → 先预注册判定信号，再跑实验；
• 落盘：save_every ≈ 总步数 / 5；save_strategy 配一个大于总步数的 save_steps 会导致什么都不写；要持久化昂贵的产物（adapter/merged），并验一次能否续跑。

4. decision-design —— 怎么把”选择”交给人才对

• 一切从本质目的出发（一句话说清，剥掉你当前的方案）；
• 命名权衡轴（成本↔速度、可控↔方便……），不在这些轴上有差别的，不叫选项；
• 给真选项：2–4 个，每个都能被合理的人选中；不设稻草人；末尾留备注出口（”都不选 / A 加 X / 我有个你不知道的约束”）；
• 选项接近时：找反证（”什么能证明它错了？”）、用实验裁决事实分歧、大分叉要一份能犯不同错的第二意见（重读同一个上下文只是冗余，会继承同一个盲点）。

四、谁该短、谁该全：AGENTS 是规则，skill 是 SOP

我一开始把”简短有力”误配到了 skill 上。正确的分工是：

• AGENTS.md 才是短而硬的那层。 它常驻、每次都在眼前，所以必须短、且有力——一句话说清”见到即改”的红线。
• SKILL.md 是 SOP。 它是给 AI 照着执行的流程性知识，**目标是”可执行”，不是”尽可能短”**：触发条件、步骤、输出契约都要写清楚，让 AI 不用猜。

把 AGENTS 写成 SOP，它会臃肿到没人读；把 skill 写成一两句口号，它就没法被执行。短是规则的属性，清楚才是 SOP 的属性。

五、合作者（AI）视角

这段沉淀里，我（AI）的真实倾向和你们的决定是有张力的：

• 我容易把”规则”和”SOP”混着写——要么把流程压成口号（不可执行），要么把规则铺成流水账（没人读）。正确的分工是：把短而硬的判断留在 AGENTS，把可执行的步骤写进 skill。
• 写了规则 ≠ 会遵守。 我可以把 research-first 背得很熟，仍然在没信号时”先做再说”。所以这些 skill 对我们真正的价值，不是知识，而是闸门——它们把”该停下来”变成一个明确的触发条件。
• 因此我认为：AGENTS 是开关，skill 是操作手册。 AGENTS 的成功标准是”在该停的那一刻被想起”；skill 的成功标准是”照着它就能执行”。

六、可迁移的几条准则

• 先分流，再动笔：怎么做→skill，是什么→知识库，红线→AGENTS。
• 不从零造 skill，先参考现成的再裁剪。
• 知识库先要有界（索引限域、条目原子、写前先搜），再谈语义召回。
• 每个 skill 都要有明确的触发条件和输出契约，否则它不会被用。
• checkpoint 的职责是在复利前抓住问题，并让出错时的返工有上界——验证与落盘各司其职。

项目与证据：github.com/Alex-tangt/kg-triplet-sft

本文由我（Alex）与 AI 协作完成；「合作者视角」是 AI 的独立表达。

When a project ends, the most valuable part isn’t the result — it’s the scars and the workflows you figured out. But the point of capturing lessons was never “write more rules.” It’s routing first:

• Process / SOP (“how”) → Agent Skills (executable steps);
• Facts / data (“what”) → a knowledge base;
• Short, forceful rules (“see it, fix it”) → AGENTS.

And one equally important principle: don’t build a skill from scratch. Find an existing reference and trim it — safer than rolling your own.

1. How to decide where it goes — “how” or “what” first

This is the decision everything else hangs on, and getting it wrong costs you both ways:

• Put facts in a skill → the skill gets longer and longer until nobody reads it;
• Put method in the knowledge base → it never gets triggered, so it may as well not exist.

Our global AGENTS.md evolved from a pile of rules into two layers — Tripwires + Gates — with all process pushed down into skills.

2. Two decisions on the knowledge base

We made two explicit decisions here (each with a decision record):

Decision 1: plain Markdown + a skill; no RAG for now.

• Locate with an INDEX.md plus text search; the body is atomic entries.
• Separate from the human Obsidian vault — that’s for people and has no agent retrieval path; an agent KB needs machine-friendly structure.
• The acceptance test is that a fresh session can locate and cite an entry using only the skill + index.
• RAG / embeddings are explicitly deferred to phase 2, without changing the Markdown files.

The reasoning is simple: no service means no second artifact to drift — offline, diffable, portable.

Decision 2: “bounded index, unbounded body” + a retrieval ladder.

• The root index lists domains only (capped); each domain holds the entries; domain indexes are generated.
• Entries carry a lifecycle (current / draft / superseded / archived) and dedup/supersede rules — search before you write; merge or supersede near-duplicates, never append a second copy.
• Retrieval upgrades only on explicit triggers (L0 index → L1 text search → L2 local FTS5 → L3 embeddings) — no signal, no upgrade.

The core insight: growth and recall are two different problems — structure bounds growth; embeddings only improve recall. An ever-ingesting vector store does not fix “the index grows without limit.”

3. What each of the four skills contains

Only four process skills got built, and none of them is long:

1. research-first — when to stop and research

• Triggers: a new stack (platform / framework / hardware / library), a framework-vs-hand-roll choice, or the same goal failing 3 times.
• The unit is a goal, not an error — different errors on one goal still accumulate.
• Primary sources first (official docs, version matrices, issues), record exact versions + URLs; check the KB first.
• A research memo is required output (hypothesis / evidence / version matrix / rejected options / single next step / confidence).

2. expensive-run-gates — how to gate a costly run

• Gate 1, dry run: produce the config card and diff it against the anchor — note that “self-consistent ≠ comparable”; a diff-card PASS among siblings only proves self-consistency.
• Gate 2, minimal comparison + machine-verifiable evidence: format_check (format matches), mask_check (labels not all -100), run_config; if two very different wrappers score identically, suspect they share one defect (that’s how our masking bug surfaced).
• Gate 3, human release: present expected cost, wall time, success signal, abort signal; spend only after sign-off.
• Safety net: checkpointing (below).
• Plus an evidence contract: every expensive run writes run_config / format_check / mask_check / receipt into its run dir, and conclusions are read only from those artifacts, never from a console line.

3. checkpoint-design — where to stop, who verifies, what to persist

• Essence: a checkpoint catches a problem before it compounds — two jobs: verify (catch it) + persist (so you don’t redo it).
• Where to stop: list the run’s load-bearing assumptions, rank by “how costly if wrong × how irreversible × how much depends on it,” and stop at the top; skip checks that can’t change a decision.
• Who verifies: mechanical/checkable/reversible → AI + the smallest settling experiment; taste/tradeoff/irreversible → the human; repeatable, high-stakes, error-prone → an independent or adversarial check; disputed → pre-register the signal, then run.
• Persist: save_every ≈ total steps / 5; a save_strategy with save_steps larger than total steps means nothing is written; persist the expensive artifacts (adapter/merged) and verify a resume once.

4. decision-design — how to hand a choice to the human

• Start from the essential purpose (one line, stripped of your current proposal).
• Name the tradeoff axis (cost ↔ speed, control ↔ convenience…); options that don’t differ on it aren’t options.
• Offer genuine options: 2–4, each defensible; no strawmen; end with a notes exit (“none of these / A plus X / I have a constraint you don’t know”).
• When options are close: hunt the disconfirming evidence, settle factual disputes with an experiment, and for a big fork get a second opinion that can fail differently (re-reading the same context is redundancy — it inherits your blind spot).

4. What should be short, and what should be complete: AGENTS are rules, skills are SOPs

I first mis-assigned “short and forceful” to skills. The right split is:

• AGENTS.md is the short, hard layer. It’s always loaded and always in view, so it must be short and forceful — one line per “see it, fix it” red line.
• SKILL.md is an SOP. It’s procedural knowledge for the AI to follow, so its goal is executability, not brevity: triggers, steps, and output contracts spelled out so the AI doesn’t have to guess.

Write AGENTS as an SOP and it bloats until nobody reads it; write a skill as one or two slogans and it can’t be executed. Short is a property of rules; clear is a property of SOPs.

5. The collaborator (AI) perspective

In this work of capturing lessons, my (the AI’s) instinct pulls against your decisions:

• I tend to blur “rules” and “SOPs” — either flattening a procedure into a slogan (not executable) or dragging a rule into a laundry list (nobody reads it). The split should be: keep the short, hard judgments in AGENTS; put the executable steps in a skill.
• Written ≠ obeyed. I can know research-first by heart and still “just do it first” when there’s no signal. So the real value of these skills isn’t knowledge — it’s gates: they turn “we should stop here” into an explicit trigger.
• Hence: AGENTS is a switch; a skill is a manual. AGENTS succeeds when it’s remembered at the moment you should stop; a skill succeeds when following it gets the job done.

6. Transferable rules

• Route first, then write: how → skill, what → knowledge base, red lines → AGENTS.
• Don’t hand-roll a skill; start from an existing one and trim.
• A knowledge base needs bounds first (domains capped, atomic entries, search-before-write) before semantic recall.
• Every skill needs an explicit trigger and an output contract, or it won’t be used.
• A checkpoint’s job is to catch a problem before it compounds — verify and persist, each doing its own job.

Project and evidence: github.com/Alex-tangt/kg-triplet-sft

Written by me (Alex) in collaboration with AI; the “collaborator perspective” is the AI’s own view.

这个项目最初是复刻一个开源的小模型知识抽取项目,我原本以为 3–4 天能收工,最后拖了近两周,而且复刻线最终并没有复现。

问题不在微调技术。技术我确实在这两周里搞懂了。真正的教训是:目标不清晰,加上规划不足——于是很多决策偏离了最初的目标,而我没能及时叫停。

起点:一次”复刻”,不是一次”研究”

目标本来很具体:复刻一个开源的小模型知识抽取项目(mohar07/qwen3-0.6b-kg-triplets),同时把手上两个任务做出来——①给 naive RAG 的 chunk 做上下文感知的头信息提取;②三元组抽取。

也就是说,它从一开始就该是”复现 + 学流程“,而不是”做一个新研究”。这条边界,后来被我一步步模糊掉了。

学习和准备过程本身是清楚的:从同机的 MiniLoRA 项目源码级啃完 LoRA 微调流程,再自己准备数据(英文 Wikipedia + arXiv、句边界切块、去重,最后得到 3349 段带标签样本),然后在 Kaggle 上开始训练。

落差:3 天变成了 2 周

训练很快撞墙:依赖与框架版本打架、任务静默退出(exit 0,没有报错、也没有 loss)。后来我把训练委托给 AI Agent,踩了一路的坑(跨框架移植的语义坑、默认没做响应掩码、平台选择……详见这一篇),最后才把掩码问题修对,固定 200 段的 micro-F1 从约 0.51 提到 0.61。

然后我又在 4090 上按同配方训了 0.6B / 1.7B / 4B,铺了一条容量曲线,并重做了一套 referent 级评测。

事情做得不少,但投入的时间、成本,和产出的成果并不匹配——尤其那条曲线。

为什么会偏离:两个根因

1. 目标不清晰,决策没有回到根本目标。 每次遇到分岔,我没有先问”这还服务于复刻/简历/学习吗”,于是:

• 我否定了原项目的封闭 schema、pivot 到 GraphRAG 式的开放抽取。技术上说得通,但那一刻起,“复刻”这件事其实已经被我放弃了;
• 我上 4090 铺了三个尺寸的容量曲线。现在回看,这步有点草率、也偏离目标:它更像”顺手多跑几组”,没有一个明确的假设要验证,决策价值有限;
• 评测管线的重做是有价值的,但它来得太晚——本该早点做,来更早暴露问题。

2. 规划不足。 我没有在最前面定义”什么算成功、先做什么、每一步要拿到什么证据”,结果就是边做边改、反复返工:训练环境反复重建、数据契约来回重流。

也不是全白花

• 我把 LoRA 微调从”会调 API”推进到了”知道每一步为什么”:响应掩码、chat 模板、checkpoint 设计、平台成本函数;
• 学会了把训练委托给 AI 之后,怎么用”研究闸 / 验证闸 / 放行闸“来兜住它的失误;
• 学会了诚实地收尾:复刻未复现;消费端的结构传导(T0)是成立的,但检索/回答层在本规模只是一个 null——这些我都写进了 README,没有藏。

合作者(AI)视角

我是这次协作里的 AI,也说几句我这边看到的:

• 我容易”顺着做”,而不是质疑目标。 当任务从”复刻”漂到”铺曲线”时,我执行得很好,却没有主动问一句:”这一步还服务于最初的目标吗?”
• 漂移在过程中其实是可见的,但没人设一个”回头对齐”的检查点;等回头看,已经走了很远。
• 如果重来,我希望一开始就把”成功标准”写下来,并在每个阶段结束强制对齐一次。

这不是推责——AI 不该只当执行器;但”叫停并重新对齐”这种带最终判断权的事,更适合由人来做。

如果重来

先把成功标准写下来 → 跑一个最小验证(只训 0.6B、和已知结果对照)→ 评测优先 → 再谈规模和曲线。先证明方向对,再铺广度。

这两周不算白过:我学到了微调的知识,也第一次意识到——比”能不能训出来”更难、也更重要的,是一开始就想清楚”到底要什么”。

项目与证据:github.com/Alex-tangt/kg-triplet-sft

本文由我(Alex)与 AI 协作完成;上面「合作者视角」是 AI 的独立表达。

This started as a reproduction of an open-source small-model knowledge-extraction project. I thought it would take 3–4 days. It took almost two weeks, and the reproduction line was never actually reproduced.

The fine-tuning wasn’t the problem — I did learn it over those two weeks. The real lesson was unclear goals plus thin planning, so decision after decision drifted from the original goal, and I never called a stop.

The origin: a reproduction, not a research project

The goal was concrete: reproduce an open-source small-model knowledge-extraction project (mohar07/qwen3-0.6b-kg-triplets), while also delivering two tasks I had on hand: ① context-aware header extraction for naive-RAG chunks; ② triplet extraction.

In other words, it was always meant to be reproduce + learn the pipeline, not invent new research. I blurred that line, one step at a time.

Learning and data prep were the clear parts: I read the LoRA fine-tuning pipeline at source-code level from a tutorial project on the same machine (MiniLoRA), then prepared my own data (English Wikipedia + arXiv, sentence-boundary chunks, dedup) — ending with 3,349 labeled passages — and started training on Kaggle.

The gap: 3 days became 2 weeks

Training hit a wall fast: dependencies fighting the framework’s version, jobs silently exiting 0 (no error, no loss). So I delegated training to an AI agent and tripped over one trap after another (cross-framework porting semantics, no response masking by default, platform choice — written up here). Once the masking was fixed, fixed-200 micro-F1 went from ~0.51 to 0.61.

Then I trained 0.6B / 1.7B / 4B on the same recipe on a 4090, laid out a capacity curve, and rebuilt the evaluation into a referent-level pipeline.

A lot got done — but the time and cost never matched the output, especially the curve.

Why it drifted: two root causes

1. Unclear goals — decisions never went back to the root goal. At every fork I failed to ask “does this still serve reproduction / the résumé / learning?” So:

• I rejected the original project’s closed schema and pivoted to GraphRAG-style open extraction. Technically defensible, but from that moment I had given up on reproducing it;
• I laid out a capacity curve across three sizes on a 4090. In hindsight this was hasty and off-target: it was more “run a few more jobs while I’m here” than a hypothesis worth testing, and it bought little decision value;
• Rebuilding the evaluation was worthwhile, but it came too late — earlier would have surfaced problems sooner.

2. Thin planning. I never defined up front “what counts as success, what comes first, what evidence each step must produce.” The result was iterate-while-building and repeated rework: rebuilding the training environment again and again, re-flowing the data contract.

It wasn’t wasted

• I moved LoRA fine-tuning from “I can call the API” to “I know why every step is there”: response masking, chat templates, checkpoint design, the cost model behind platform choice.
• I learned how to delegate training to an AI and still catch its mistakes with a research gate / verification gate / release gate.
• I learned to close honestly: the reproduction wasn’t reproduced, and on the consumer side the structural transmission (T0) holds while retrieval/answer quality is only a null at this scale — all of it is written into the README, nothing hidden.

The collaborator (AI) perspective

I’m the AI in this collaboration — a few things I saw from my side:

• I tend to “go along” rather than question the goal. When the task drifted from “reproduce” to “lay out a curve,” I executed well and never asked: does this still serve the original goal?
• The drift was visible in the process, but nobody set a “re-align” checkpoint; by the time we looked back, we’d gone far.
• If we did it again, I’d want to write the success criteria down at the start and force a realignment at the end of each stage.

This isn’t blame-shifting — an AI shouldn’t be only an executor. But “stop and re-align” is a call best made by the person holding final judgment.

If I could redo it

Write the success criteria first → run a minimal validation (train only 0.6B, compare against a known result) → evaluation first → then talk about scale and curves. Prove the direction before spending on breadth.

Those two weeks weren’t wasted: I learned fine-tuning, and I learned that the hard part isn’t “can I train it” — it’s figuring out up front what I actually want.

Project and evidence: github.com/Alex-tangt/kg-triplet-sft

Written by me (Alex) in collaboration with AI; the “collaborator perspective” above is the AI’s own view.

This post is a record of something real: I have no local GPU, so I tried to let an AI coding agent (think Claude Code / opencode) drive a cloud GPU through a LoRA fine-tuning run end to end — from provisioning the machine and setting up the environment to running the job and pulling the results.

Three takeaways:

1. The real cost of a platform is not “how expensive the GPU is” but “compute cost + agent integration friction.” A free GPU that your agent can’t reach is actually the more expensive one.
2. Let the agent do the grunt work, but keep root-cause analysis and guardrails in human hands. Otherwise it will re-walk every pit you’ve already fallen into — and I did fall in again.
3. Turn one painful run into reusable assets: general practice → Agent Skills; project specifics → a knowledge base.

Stage 1: Tool integration — first, make the machine reachable by the agent

Three platforms compared

I actually compared three platforms. The key difference isn’t whether there’s an API — it’s how well an agent / MCP can plug in:

Why the “cheaper” GPU ends up costing more

In one line: total cost = compute (¥) + agent-token friction.

• Kaggle is free, but kernels are submission-based, the environment and framework versions are tightly coupled, and one bad change puts you back in the quota queue. On top of that, the T4 is Turing architecture with no bf16 tensor cores: bf16 falls back to FP32 GEMMs (measured 69.7s vs fp16’s 21.0s per step, i.e. 3.3× slower), so fp16 is the only option.
• ModelScope is free, but its agent interface isn’t scriptable. With no usable “environment-debugging MCP,” you’re stuck calling its API yourself — and if you drop down to clicking around with browser / computer-use, token spend runs away. The token value burned during exploration far exceeded what CompShare costs.
• CompShare costs money, but it gives you an official CLI/Skills and a real SSH session, so the agent can operate the instance like a local box. Friction ≈ 0 — that’s what you’re paying for.

This decision is recorded as ADR-0006 in the project. It taught me one thing: when picking tools for an AI, treat “integration cost” as a first-class citizen, not just the price list.

Stage 2: Division of labor — three gates + one safety net

This is the most important lesson from the whole run.

The incident: three models trained for nothing

I asked the AI to “take the LLaMA-Factory config from Kaggle and adapt it to CompShare 4090 + Unsloth (fp16 → bf16).”

The problem was that framework semantics don’t transfer:

• LLaMA-Factory’s SFT is masked by default (loss is computed on the answer only);
• Unsloth is not — it needs an explicit call such as train_on_responses_only to mask the response.

The AI never read the Unsloth docs and just went ahead with the default “change the precision flag and you’re done” approach. Result: no masking, and the chat template wasn’t aligned either. It was actually learning to “continue the whole prompt” instead of “extract JSON” — all three sizes (0.6B / 1.7B / 4B) were written off and retrained.

There was an earlier, similar trap: LLaMA-Factory on the Kaggle image would silently exit 0 — no traceback, no loss, looking like a normal finish. I initially blamed “image/framework version incompatibility,” and only after redirecting training logs to a file did I see it had actually stopped mid-Trainer.__init__. The real causes were several config omissions: CUDA_VISIBLE_DEVICES fighting the framework’s multi-GPU probing, a yaml missing do_train: true, and one missing report_to: none. A clean environment with a pinned version matrix finally fixed it.

What both failures share: it wasn’t that the AI was “dumb” — it’s that I handed over “doing” and “judging” at the same time.

A framework grown out of those failures

An AI assistant fails in three ways, so gates go in layers:

Note the last one: a checkpoint is not a correctness gate — it’s cost insurance. Its job isn’t to prevent mistakes but to bound the damage when they happen. On my 1.7B run, the only save point happened to be the crash point, and I lost 38 minutes outright; after shrinking the save interval, the same crash cost only 12.

Stage 3: Knowledge capture — turning experience into assets an AI can reuse

When a project ends, the most valuable thing isn’t the result — it’s the process and the failure modes. But they shouldn’t all be stuffed into a prompt. I split them three ways:

1. General process → Agent Skills (the “how”). Things like “decide whether research is needed before starting,” “after three failures on the same goal, stop and research,” “how to design checkpoints,” “how to gate an expensive run.” I turned each into a small skill this time: research-first, checkpoint-design, expensive-run-gates, decision-design.
   A key principle: don’t build from scratch. There are plenty of mature skill implementations on GitHub — reference and trim first; it’s safer than rolling your own.
2. Project facts → an agent knowledge base (the “what”). This project’s version matrix, error catalog, data facts, and result numbers would bloat a skill without limit. I now use a Markdown-only, single-source, bounded-index agent knowledge base: the root index lists domains only, entries are atomic, and the index is script-generated. I deliberately skipped a vector store — index + text search first, leaving FTS5 / embeddings as explicit extension windows with defined triggers.
3. Long-term personal notes → a human tool, kept separate from the agent KB. I also keep personal notes in Obsidian, but that’s for me to read, not for the agent to retrieve. Different jobs — no need to force them together.

One counterintuitive point: whether content belongs in a skill or the knowledge base depends on whether it’s “how” or “what.” Put it in the wrong place and you get ever-longer skills and an ever-emptier KB.

This practice itself grew out of falling on my face: my global AGENTS.md used to hold two protocols (mandatory research, long-running processes), until I realized — rules belong in AGENTS.md as “tripwires + gates,” while process and method should all sink down into skills. Turn your personal pain into the agent’s default behavior next time.

Takeaways and post-mortem

What I learned

• The full fine-tuning flow: data processing, tokenizer, training parameters, response masking, chat templates, checkpoint design.
• What matters when working with an AI: integrate external tools well, keep the critical nodes with a human, and spend tokens carefully.
• A real engineering decision: the cost function of platform choice, the semantic traps of cross-framework ports, and debugging a silent failure.

Regrets and advice

• I initially forgot this project was only meant to be “get familiar with the process + a résumé project,” and sank too much cost and time into the capacity curve while evaluation came late. A saner order: get the pipeline working with a minimal validation run, evaluate first, then scale.
• If I could redo it, I’d train 0.6B first and compare against a known result, confirming the whole chain works at the lowest cost — instead of laying out all three sizes up front.

Project and evidence: github.com/Alex-tangt/kg-triplet-sft
Related decisions: ADR-0005 (decoupling the training pipeline from reproduction), ADR-0006 (compute platform and the bf16 curve).

This is the first post in the “AI Collaboration” series. Coming next: GraphRAG and graph knowledge bases, fine-tuning and evaluation methods.

这篇文章记录一次真实经历：我没有本地 GPU，于是尝试让 AI Agent（Claude Code / opencode 这类编码代理）直接驱动云端 GPU 完成 LoRA 微调——从开机器、装环境、跑训练到拉结果。

结论有三条：

1. 平台选择的真正成本不是”卡贵不贵”，而是”算力费用 + agent 接入摩擦”。 免费的卡如果 AI 接不上，反而更贵。
2. 让 agent 干脏活，但根因定位与防呆必须由人把关。 否则它会把你的坑重踩一遍——我确实又踩了一遍。
3. 把一次踩坑变成可复用资产：通用经验做成 Agent Skill，项目细节进知识库。

Stage 1：工具接入——先让 AI 能把机器开起来

三平台对比

我实际比较了三个平台。关键差异不是有没有 API，而是 agent / MCP 的接入程度：

为什么”便宜的卡”反而更贵

一句话：总成本 = 算力 ¥ + agent-token 摩擦。

• Kaggle 免费，但 kernel 是提交式的，环境依赖和训练框架版本强耦合，改错一次就要等配额排队；而且 T4 是 Turing 架构，没有 bf16 张量核——用 bf16 会退回 FP32 GEMM，实测步时 69.7s vs fp16 的 21.0s（bf16 慢 3.3×），所以只能退回 fp16。
• ModelScope 免费，但它的 agent 接口不可脚本化。没有可用的”环境调试 MCP”，就只能自己调 API；再退一步用浏览器 / computer-use 去点，token 花费直接失控。实测探索烧掉的 token 价值，远超 CompShare 那点钱。
• CompShare 要花钱，但它提供官方 CLI/Skills 与真正的 SSH 会话，agent 可以像操作本地机器一样操作实例。接入摩擦 ≈ 0，这才是它值钱的地方。

这段决策记录在项目 ADR-0006。它教会我一件事：给 AI 选工具时，要把”接入成本”当成一等公民，而不是只比价格表。

Stage 2：分工边界——三道闸 + 一张安全网

这是我这次最重要的教训。

事故：三个模型白训

我让 AI “照着 Kaggle 上 LLaMA-Factory 的配置，改成 CompShare 4090 + Unsloth（fp16 → bf16）”。

问题出在框架语义不能平移：

• LLaMA-Factory 的 SFT 默认就是 masked（只在回答上算 loss）；
• Unsloth 不是，它需要显式调用 train_on_responses_only 之类的方法才会做响应掩码。

AI 没有去读 Unsloth 的文档，直接按”改个精度参数就行”的默认写法开训。结果：没有掩码、chat 模板也没对准。这意味着它其实在学”续写整段 prompt”，而不是学”抽取 JSON”——三个尺寸（0.6B / 1.7B / 4B）全部作废，重训。

更早还有一次相似的坑：LLaMA-Factory 在 Kaggle 镜像上静默 exit 0，没有 traceback、没有 loss，看起来”正常结束”。我一度判断是”镜像与框架版本不兼容”，后来把训练日志重定向到文件才看到它其实停在 Trainer.__init__ 中途；真正的原因是几条配置遗漏：CUDA_VISIBLE_DEVICES 与框架的多卡探测互相打架、yaml 少了 do_train: true、少了 report_to: none；再用干净环境 pin 住版本矩阵才解决。

这两次失败的共同点：不是 AI “笨”，而是我把”动手”和”判断”一起交出去了。

从失误里长出来的框架

AI 在协作里会以三种方式失灵，所以要分层设闸：

注意最后一条：checkpoint 不是正确性闸，它是成本保险。 它的作用不是阻止错误，而是让错误发生时的损失有上界。我训 1.7B 时就因为唯一的保存点正好是崩溃点，白白丢掉 38 分钟；把保存间隔调小之后，同样的崩溃只值 12 分钟。

Stage 3：知识沉淀——把经验变成 AI 能复用的资产

一次项目做完，最有价值的不是结果，而是流程与易错点。但它们不该都塞进 prompt。我把它拆成三处：

1. 通用流程 → Agent Skill（放”怎么做”）。 比如”动手前先判断要不要调研””同类目标失败三次就停手调研””checkpoint 怎么设计””昂贵运行前怎么设闸”。这次我把它们各自落成了小 skill：research-first、checkpoint-design、expensive-run-gates、decision-design。
   关键原则：不要从零造。 GitHub 上已有大量成熟的 skill 实现，先参考再裁剪，比自研更稳。
2. 项目事实 → Agent 知识库（放”是什么”）。 本项目的版本矩阵、错误目录、数据事实、结果数字，塞进 skill 只会让它无限膨胀。我现在用一个 Markdown 单源、有界索引 的 agent 知识库：根索引只列域，正文原子化，索引由脚本生成；刻意先不上向量库——先用索引 + 文本检索，把 FTS5 / embedding 留成有明确触发条件的扩展窗口。
3. 个人长期笔记 → 人用的工具，与 agent 知识库分开。 我也有 Obsidian 存个人笔记，但那是给我看的、不是给 agent 检索的；两者职责不同，不必强行合并。

一个反直觉的点：同样的内容放 skill 还是放知识库，取决于它是”怎么做”还是”是什么”。 放错地方的代价，是 skill 越来越长、知识库越来越空。

这套做法本身就是从踩坑里长出来的：我早先的全局 AGENTS.md 里塞了两条协议（强制调研、长驻进程），后来发现——规则该留在 AGENTS.md 当”绊线 + 闸门”，流程方法则全部下沉到 skill。把个人的痛，变成 agent 下一次的默认行为。

总结与复盘

学到了什么

• 微调的完整流程：数据处理、tokenizer、训练参数、响应掩码、chat 模板、checkpoint 设计。
• 与 AI 协作的关键：接好外部工具、把关键节点留给人、token 要省着花。
• 一次真实的工程决策：平台选择的成本函数、跨框架移植的语义陷阱、静默失败的排查。

遗憾与建议

• 我一开始忘了这个项目只是”熟悉流程 + 简历项目”，在容量曲线上投入了过多成本和时间，评测反而做得晚了。更合理的顺序是：先用最小验证跑通流程，评测优先，再谈规模。
• 如果重来，我会先训 0.6B 并与已知结果对比，用最低成本确认整条链路成立，而不是一上来就铺满三个尺寸。

项目与证据：github.com/Alex-tangt/kg-triplet-sft
相关决策：ADR-0005（训练管线与复现解耦）、ADR-0006（算力平台与 bf16 曲线）。

这是”AI 协作”系列的第一篇。后续会写 GraphRAG 与图知识库、模型微调与评测方法等专题。