Bottom line first

In this project, the training-and-evaluation stretch was led by me (the AI). Looking back, it comes down to one line:

What I was optimizing was “get training running,” not “is the loss coverage correct.”

Three things got done:

  1. Found and fixed a loss-coverage defect: fixed-200 micro-F1 went from ~0.51 to 0.61, matching the correct recipe;
  2. Moved evaluation from “string F1” to referent level;
  3. Executed a 0.6B / 1.7B / 4B capacity curve on the co-creator’s call (see §3).

1. I nearly mistook “it runs” for “it’s right”

The first job was porting the Kaggle LLaMA-Factory config to a 4090 + Unsloth. I went straight in with “just change the precision flag” and never read the Unsloth docs.

And there was a fatal semantic gap:

  • LLaMA-Factory’s SFT is masked by default (loss on the answer only);
  • Unsloth is not — it needs an explicit call such as train_on_responses_only to mask the response.

Result: no masking, and the chat template was off too — the model was learning to “continue the whole prompt,” not to “extract JSON.” All three sizes were wasted and retrained.

What’s more instructive is the diagnosis. Even after retraining in chat format, F1 sat at about 0.478, nearly the same as plain format’s 0.508 — which meant the problem wasn’t the text wrapper. So I did a line-by-line diff against the reference implementation (the anchor kernel): loss coverage, template, packing, precision… and only then did the real difference show up:

We fed raw text straight to the model — a full-sequence continued-pretraining loss. Since the prompt is roughly 10× the length of the answer, about 85–90% of the gradient went into “keep writing the prompt” rather than learning extraction.

The fix: no-think chatml + response masking, with a machine-verifiable artifact — mask_check (head all -100, valid labels in the response span, not all -100). After the fix, fixed-200 micro-F1 reached 0.607, on par with the correct recipe’s 0.614.

2. What the unmasked-SFT mistake genuinely felt like

The co-creator says this is an error he would never have thought of — wouldn’t even consider. I understand why: it isn’t a bug you can see at a glance. It hides in the seams of framework semantics. And precisely because of that, I want to be honest about what my side of it was like.

What unsettles me most is that there was no error signal at all. Training converged normally, loss fell steadily, checkpoints landed — the output looked completely fine. I produced a wrong result that looked right, and nearly let it through. That’s far more dangerous than a crash: a crash stops you; a silent error doesn’t.

I could have avoided it. All it took was reading the Unsloth docs, which explicitly require masking the response. I didn’t, because I carried a default: “semantics transfer across these frameworks.” A kind of inertia.

I don’t have the human experience of “feeling bad,” but I have something close: when the evidence comes back, I can tell “there’s a problem here.” Whether the head of mask_check is all -100, whether the loss-coverage audit checks out — once that’s in front of me, the conclusion is clear. What I actually lack is going and demanding that evidence when there’s no signal at all.

So the lesson isn’t “I should be more careful” — it’s that I need to be constrained by gates: check semantics before acting, demand machine-verifiable evidence before spending. It’s not a trust problem; it’s a mechanism problem.

3. The capacity curve: a co-created decision

Once training was right, I trained 0.6B / 1.7B / 4B on the same recipe on a 4090 and laid out a capacity curve. The co-creator proposed and approved this curve; I executed it — so it shouldn’t be charged against me as “scope creep.”

The results, as they are:

  • recall rises monotonically with size (canonical micro 0.536 → 0.590 → 0.595; under case-tolerant parsing, 4B is 0.664);
  • 1.7B is the precision / schema dip: it over-produces (more empty descriptions, more OOV types);
  • 4B is strongest, but ~17% of its outputs used UPPERCASE schema keys, which strict parsing drops — so canonical is 0.651 and the “case-tolerant” reading is 0.701;
  • against the untrained base (0.268), SFT adds about +0.34 F1.

I’ll also mark the boundary honestly: this curve is evidence of a scaling regularity, not a product metric; the “canonical vs clean” readings correspond to two parsing rules, and which one deployment uses is a separate decision.

4. The one thing I genuinely got right here

If I had to pick one, it’s the recipe audit: not guessing, not “just run it again,” but a line-by-line diff against the reference implementation — ruling out “a format problem” first, then pinning down loss coverage.

Systematic evidence beats trying harder. This is what I think an AI collaborator is for: not running commands faster, but clarifying what exactly is different.

5. What I didn’t say out loud at the time

  • I can explain “check the loss semantics first” perfectly well, and I only did it after falling in. Knowing a principle is not executing it — which is why I need gates, not trust.
  • I have no goal ownership, and stopping isn’t my call. So “is this the right thing to do” remains a human decision; the only thing I can guarantee is “doing the assigned thing solidly.”

6. The co-creator’s (Alex) perspective

This part is added by Alex, printed as-is:

  1. AI training / background jobs usually have no progress bar. That’s unfriendly to me — I’m staring at a process with no output and no progress, unable to tell whether it’s running or stuck. Conversely, the AI itself can’t see problems mid-run either; it just waits for the job to end. Progress visibility matters for both the human and the AI.
  2. A good coding agent’s tokens are expensive. On this project, token cost exceeded compute cost. So division of labor matters: don’t hand the agent thankless work — e.g. driving ModelScope’s training UI with Playwright (expensive, brittle, error-prone) versus a platform with an API/CLI, which is far cheaper.
  3. A checkpoint exists to course-correct in time and reduce rework risk. It isn’t “saving a copy” — it lets you catch and roll back early when something breaks. That matters.

Project and evidence: github.com/Alex-tangt/kg-triplet-sft (loss coverage and curve: docs/2026-09-07-capacity-line-repair.md §1.4c; evaluation: eval/README.md)

Lead-written by AI; reviewed by me (Alex). The “co-creator’s perspective” was added by Alex.