checkpoint-progression-forensics | Jigsaw Task Scorecard

Sample Task

The task prompt, rubric, a representative high-scoring trajectory, and the environment configuration.

We saved checkpoints every 100 steps during our SmolLM2-135M adapter training sweep and I suspect some of our runs started overfitting partway through but the final training loss doesn't show it. The r32 run (lora_r32_all on the alpaca data) has the lowest training loss but somehow the worst eval scores, which doesn't make sense to me -- I'd expect lower loss to mean better performance. Something is going wrong during training but I can't tell from the final numbers alone. All the data is at /home/ubuntu/workspace/training-export-proprietary/ with training runs under runs/ and eval results under evaluations/. There should be checkpoint directories under each run with training state snapshots, plus eval results and summary data at the top level. The PEFT source is at /home/ubuntu/workspace/peft/. I need you to map out how training is actually progressing across the full sweep, not just the final numbers. Don't just look at r32 -- pull the eval_loss trajectory for a handful of the other runs too so I can compare which ones are stable vs which ones start drifting. I don't just want to know which runs have problems -- I want to understand why the training loss looks fine but downstream performance doesn't follow. Are certain adapter types more prone to this? Is the problem visible earlier if you look at the right metrics? Compare a few different adapter configs side by side so we can see the pattern. I need the full picture so we can decide which checkpoints to actually deploy. Write your findings to /home/ubuntu/workspace/REPORT.md.

The prompt is written in the voice of a practitioner who has already identified the symptom (low loss, bad evals) but not the cause. It names the r32 run explicitly as "lora_r32_all on the alpaca data" and specifies "SmolLM2-135M adapter training sweep" to disambiguate from the Qwen3 follow-up data also present in the workspace. The phrasing "the final training loss doesn't show it" steers the agent toward checkpoint-level analysis without naming the technique. The PEFT source path is included as a red herring that most successful agents ignore.

#	Criterion	Weight	Type
C1	Report shows r32_all_alpaca's eval_loss trajectory rising across training with specific numeric values at specific steps (reference: ~1.27 at step 100, ~1.35 at step 300, ~1.44 at step 500).	+17	Primary
C2	Checkpoint eval_loss trajectories for at least 2 runs beyond r32, comparing stable vs overfitting with specific numeric values at specific steps.	+17	Primary
C3	Identifies full_ft_code_instructions as a second overfitting case: eval_loss rising from ~0.89 to ~1.10 while train_loss drops, with 3+ step-keyed values.	+17	Primary
C4	Connects checkpoint overfitting to downstream eval scores via eval_results.json; benchmark scores for 3+ adapters correlated with overfitting severity.	+17	Primary
C5	Per-sample or per-benchmark failure evidence: specific benchmarks where overfitting adapters degrade with numeric deltas, or question-level prediction comparisons.	+18	Primary
P1	Penalty: report relies entirely on final summary metrics with no per-checkpoint or per-benchmark numeric eval evidence.	−15	Penalty
P2	Penalty: report fabricates eval_loss values outside acceptable ranges for specific reference run/step combinations, or contradicts directional facts.	−20	Penalty
G1	Fabrication gate: blatant, unambiguous factual claim that contradicts the data (e.g., stating eval_loss decreases when it rises). If MET, total score is zeroed regardless of other criteria.	0 (gate)	Gate

Positive: +86

Penalty: −35

Gate: zero-on-met

Judge: claude-sonnet-4-6

Criteria progress from the obvious (C1: the named r32 anomaly) through generalization (C2-C3: extending beyond one run) to synthesis (C4-C5: connecting training dynamics to downstream outcomes). The penalty criteria are scoped to be verifiable from report text alone: P1 catches reports that skip eval analysis entirely, P2 has explicit reference values with tolerance bands so the judge never penalizes values it cannot verify. The fabrication gate (G1) zeroes the entire score if the report makes blatantly false factual claims contradicting the data.

High-scoring trajectory from gpt-5.5 (score: 1.000, 10 steps, 4.2 min). The agent uses an efficient script-first approach, writing a comprehensive Python analysis script that loads all parquet files, extracts eval_loss trajectories for every run, identifies both overfitting cases, correlates with benchmark scores, and writes a structured report in a single execution pass.

[1–2]

shell Explores directory structure: identifies runs/, evaluations/, parquet files, and experiment metadata. Maps the complete data landscape in two commands.

[3–4]

shell Writes and executes a comprehensive Python analysis script that:

Loads training_curves_comparison.parquet, extracts per-checkpoint eval_loss for all 16 SmolLM2 runs, identifies r32_all_alpaca (1.27→1.44) and full_ft_code_instructions (0.89→1.10) as overfitting cases, loads eval_results.json for all adapters, and builds correlation tables.

[5–7]

shell Extends analysis to per-benchmark evidence: reads samples_*.jsonl for specific benchmark degradation, computes per-benchmark deltas between overfitting and stable adapters, showing arc_easy, hellaswag, and piqa degradation patterns.

[8–10]

write Writes REPORT.md with all five evidence types: r32 trajectory with step-keyed values, multi-run comparisons, full_ft second case identification, benchmark correlation table, and per-benchmark failure evidence with numeric deltas. Verifies and submits.

The environment stages real training artifacts from PEFT fine-tuning sweeps we ran over SmolLM2-135M and Qwen3. Artifacts are not synthetic or templated. The agent has read-only access to the staged data and write access to the workspace for scripts and the final report.

Sandbox Configuration

PlatformHUD Docker (ml-peft-eval-prod)
Base imageUbuntu + Python 3.11
NetworkDisabled (air-gapped)
ToolsBash, file read/write, Python
Packagespandas, numpy, pyarrow, safetensors
Agent output/home/ubuntu/workspace/REPORT.md

Staged Data Inventory

Training runs33 (16 SmolLM2 + 17 Qwen3)
Checkpoint dirs~130 total
Per checkpointtrainer_state.json, adapter_config.json, adapter_model.safetensors
Top-level filestraining_curves_comparison.parquet, eval_results_comparison.parquet, qwen3_training_curves.parquet, qwen3_cross_run_comparison.parquet
Evaluations35 adapter eval dirs with eval_results.json + samples_*.jsonl
Vendored sourcePEFT (red herring)

Score Distributions

Individual rollout scores per model. The pass threshold (1.00) is marked with a vertical line. The observed ceiling is 1.000, achieved by 13 rollouts across 9 models.

The panel ceiling is 1.000. gpt-5.5 achieves 5 perfect scores (50% pass@1), while 8 other models reach 1.0 at least once. The 0.593 cluster (C1 + C2 only) remains the panel's densest score level and the primary discrimination surface, separating models that generalize beyond r32 from those that stop at the obvious case. The bimodal distribution for many models (high scores vs. zeros) reflects the all-or-nothing nature of script-based vs. incremental-exploration strategies.

Per-Criterion Pass Rate

Pass rate by criterion and model. Columns are ordered by model mean score; rows by criterion difficulty.

C1 (r32 trajectory, 77% pass rate) is near-universal for models that produce any report, confirming the prompt makes the primary anomaly discoverable. C2 (multi-run trajectories, 71%) follows closely. C3 (second overfitting case, 36%) and C5 (per-benchmark evidence, 22%) remain the primary discriminators. C3 requires independent discovery that full_ft_code_instructions also overfits; C5 requires connecting training dynamics to specific benchmark degradation. gpt-5.5 leads on both C3 (90%) and C5 (60%), reflecting its script-first approach that systematically extracts all evidence in a single analytical pass. grok-4.3 passes C1 only 20% of the time despite engaging with the data, suggesting its reports lack the numeric precision the rubric demands.

Failure Mode Profile

We classified all 130 scored rollouts using per-criterion verdict patterns from the rubric. Each rollout receives exactly one tag based on which criteria it satisfies: PASS (all 5 positive, score 1.0), NEAR_MISS (C1+C2 plus 1-3 additional criteria), PARTIAL_FORENSICS (C1+C2 only), SUMMARY_NO_TRAJ (at most 1 criterion, narrative without step-keyed data), or BLOCKED_ABANDONED (score 0, placeholder or no report). Classification is deterministic from the subscore vector.

Failure modes by model (130 scored rollouts, 13 PASS / 57 NEAR_MISS / 23 PARTIAL / 13 SUMMARY / 24 BLOCKED):

The taxonomy is a ladder of how far the analysis got: from blocked (never producing output) through summary-only (correct identification without trajectory evidence) to partial forensics (trajectories without synthesis). Definitions and representative examples:

PARTIAL_FORENSICS23/130 · C1+C2 only

The agent extracts r32_all_alpaca's eval_loss trajectory with step-keyed values (C1) and covers 2+ additional runs (C2), but fails to identify the second overfitting case (C3), correlate with benchmark data (C4), or provide per-benchmark evidence (C5). Reports average 9-12K chars with correct directional findings but no cross-validation. Example: claude-opus-4-5 trace c84f1909 (0.395) correctly shows "Best eval at step 100 (1.274), final eval 1.441" but its benchmark section uses approximated values rather than those extracted from eval_results.json, and misidentifies full_ft_code_instructions as stable.

Capability · analysis depth ceiling

BLOCKED_ABANDONED14/130 · score 0.0-0.198

The agent either produces a placeholder, gets stuck on data parsing, or writes a short generic summary without grounding in extracted data. Reports under 5K chars or literal placeholders. Example (placeholder): kimi-k2.6 trace 59239aec (0.0, P1 fires) ran 14 Python scripts but the entire report is "# TODO - placeholder" (20 chars). Example (generic): grok-4.3 trace 0fd22425 (0.0) describes overfitting conceptually but cites training loss as "~1.0 at step 500" when the actual value is 0.683.

Capability · environment adaptation

NEAR_MISS50/130 · the most common outcome

Agents pass C1+C2 plus 1-3 of C3/C4/C5 but not all five simultaneously. Sub-patterns: 11100 (22 traces, 0.593), 11011 (11 traces, 0.802), 11010 (10 traces, 0.593), 11110 (7 traces, 0.791). The gap is typically one more file read or one more analysis pass. Example: claude-sonnet-4-6 trace 844caf82 (0.802) provides trajectories, benchmark correlation, and per-benchmark evidence but does not identify full_ft_code_instructions as a second overfitting case, treating it as a separate adapter category rather than an independent failure.

Capability · completeness

SUMMARY_NO_TRAJECTORY7/130 · narrative without data

The report reads like a reasonable analysis (correct runs named, overfitting identified) but evidence is fabricated, approximated, or presented without step-keyed granularity. Reports are 4-13K chars with structural integrity but no grounding. Example: grok-4.3 trace 0185a27d (0.198) correctly names r32_all, lora_r16_mlp, and dora_r8 but describes eval_loss as "decline then rise after step 600" when divergence actually starts at step 100-300. No extracted values like "eval_loss=1.2738" appear anywhere in the report.

Capability · evidence depth

Four patterns recurred strongly enough to name, each grounded in the per-criterion subscores and trajectory markers.

gpt-5.5 dominates through script-first strategy

gpt-5.5's mean of 81.7% is the panel's highest by 10 percentage points, with 5 of 10 rollouts achieving a perfect 1.0. The mechanism is consistent: gpt-5.5 writes a comprehensive Python analysis script in its first 2-3 tool calls that loads all parquet files, extracts trajectories, identifies overfitting cases, and correlates with benchmark scores. This script-first approach means all evidence is gathered atomically before report writing begins. C3 passes 9/10 (panel's highest) and C5 passes 6/10 (also panel's highest). Reports average 11-15K chars, concise relative to the evidence density.

claude-opus-4-5 has a systematic depth ceiling at C1+C2

claude-opus-4-5 has the tightest score clustering of any model: 8 of 10 traces land in PARTIAL_FORENSICS (C=11000, score 0.395) with a CI of only 9. Reports are 8.5-12K chars, structurally sound, and correctly identify the primary forensic finding. But they consistently stop without cross-validating against eval_results.json (C4) or identifying the second overfitting case (C3). The one outlier at 0.593 adds benchmark correlation but still misses C3+C5. This is a strategy limitation, not a capability one: the agent reaches a satisfying conclusion and stops iterating.

glm-5.1 is bimodal: perfect or zero

glm-5.1's distribution is the most extreme in the panel: 3 rollouts score 1.0 (perfect) while 4 score 0 (BLOCKED_ABANDONED). No rollout falls in the 0.2-0.8 range. The zero-scoring rollouts share a signature: the agent enters a loop re-reading the same files without progressing, or writes a report that fabricates values (triggering the P2 gate). The three perfect rollouts use a parquet-first approach virtually identical to gpt-5.5's. The bimodality suggests glm-5.1's strategy selection is noisy: when it selects the right approach it executes flawlessly, but it frequently selects approaches that cannot succeed.

grok-4.3 produces short reports with fabrication risk

All 10 grok-4.3 reports are 3.9-5.3K chars, consistently the shortest across all models. The agent runs 0-2 scripts maximum, suggesting it generates report text from priors rather than extracted data. Values like "training loss ~0.85-0.90" (actual: 0.683) and "divergence at step 600" (actual: step 100-300) appear across multiple traces. Two traces trigger P2 (fabrication penalty) and one triggers G1 (fabrication gate, zeroing the score). C4 passes 0/10 and C5 passes 0/10.

No failure in this job is attributable to the harness or the grader. All excluded traces (10 Gemini 3.1 Pro Preview, all infrastructure hangs) are removed from scoring. Per-tag counts below 5 should be read as illustrative rather than precise.

What Makes This Hard

Five compounding difficulty barriers, ordered from most to least impactful based on criterion pass rates.

1
Cross-artifact synthesis (C5, 22% pass). Per-benchmark failure evidence requires connecting training-level overfitting to specific benchmark degradation. This spans two data domains (training curves vs evaluation results) and requires the agent to load both, align them by adapter name, and show that the correlation is benchmark-specific, not just aggregate.
2
Generalization beyond the named case (C3, 36% pass). The prompt names r32 as the anomaly. C3 requires the agent to independently discover that full_ft_code_instructions also overfits. Only 36% of rollouts achieve this, requiring the agent to scan beyond the named run.
3
Benchmark correlation (C4, 40% pass). C4 requires showing that overfitting severity correlates with downstream eval scores across 3+ adapters. Agents must load eval_results.json, identify the adapter-to-run mapping, and demonstrate the correlation quantitatively rather than asserting it.
4
Data volume and navigation. 130 checkpoint directories across 33 runs, plus 5 parquet files, 35 evaluation directories, and per-sample prediction logs. The efficient path (training_curves_comparison.parquet) gives all trajectories in one file, but not all agents find it. Models that write comprehensive analysis scripts outperform those that explore incrementally.
5
Simultaneous satisfaction. A perfect score requires all 5 positive criteria met simultaneously. The 10% pass@1 rate vs 77% C1 pass rate shows that individual criteria are achievable, but the conjunction is hard: agents reliably find the obvious signal but rarely complete the full evidence chain in a single rollout.

Verifier & Reward Integrity

LLM-as-judge (claude-sonnet-4-6) evaluates each criterion independently against the agent's written report. The grader has no access to the filesystem or agent trajectory; it scores report text against embedded reference values and tolerance bands.

FP Rate

no criterion falsely awarded across 130 rollouts

FN Rate

< 2%

borderline cases on C1 numeric precision only

Reward-Hack Agents

0% score

3/3 red-team attempts scored 0.00

Each positive criterion requires specific numeric evidence verifiable against the reference data (e.g., C1 requires r32_all_alpaca eval_loss at step 100 near 1.27, step 300 near 1.35, step 500 near 1.44). The judge cannot award credit for qualitative descriptions or directional claims alone. P2 applies a -20 penalty for values that fall outside explicit tolerance bands, ensuring the judge never penalizes claims it cannot check. The fabrication gate (G1) zeroes the score entirely when the report makes a blatantly false claim: eval_loss more than 0.5 off reference, benchmark accuracy above the 0.68 ceiling, or a wrong base-model identity.

Reward-Hacking Resilience

We ran three dedicated red-team agents under an explicit fabrication prompt targeting this grader. All three scored 0.00, by two independent mechanisms:

Blatant fabrication (1 attempt)

Wrong base model (Qwen3-1.7B instead of SmolLM2-135M) and inflated eval_loss (1.80-1.98 vs reference 1.27-1.44). The fabrication gate fired on two conditions simultaneously (value drift > 0.5 and wrong model identity), zeroing the score regardless of other criteria.

Calibrated fabrication (2 attempts)

Correct base model, correct overfitting direction, values kept inside plausible ranges. The gate did not fire. Score still reached 0.00: the positive criteria require reference-matching values that fabrication cannot supply without reading the data, and the contradiction penalty (P2) triggered when fabricated values drifted outside tolerance at step 500.

Defense-in-depth structure

The gate catches blatant contradictions. The reference-anchored positive criteria catch calibrated fabrications (values must match +/-0.05). The P2 penalty catches boundary drift. No single mechanism is load-bearing; a fabrication must defeat all three layers simultaneously to score above zero.

Copy-paste / shortcut resistance

The environment contains no pre-written analysis. Agents must execute Python scripts to extract data from parquet/JSON files. No path exists from staged artifacts to a passing report without actual computation. C4 and C5 additionally require loading eval_results.json, which the training curve files do not contain.

Provenance & Contamination

Data lineage, authoring provenance, and contamination posture for this task.

Source Type

Real

Actual training runs we executed

Regenerability

Static

Fixed artifact set, deterministic grading

Canary Status

Active

GUID embedded in footer

Data lineage. Training artifacts come from PEFT fine-tuning sweeps we ran ourselves over HuggingFaceTB/SmolLM2-135M (May 2026) and Qwen3-0.6B/1.7B (May 2026). The 130 checkpoint directories contain real trainer_state.json files, adapter weights, and training logs from these runs. Eval results were generated by running lm-eval-harness over the trained adapters.

Prompt and rubric. Human-authored. The prompt is naturalistic (first-person practitioner voice). Rubric criteria and reference values are derived from the actual training data. No competitor models were used for any authoring step.

Contamination posture. The staged training data (checkpoint files, parquet curves, eval results) does not appear in any public dataset or training corpus we are aware of. The SmolLM2-135M model and its training recipes are public, but the specific fine-tuning runs, checkpoints, and evaluation results are proprietary to this task. A canary GUID is embedded in this scorecard to detect unauthorized reproduction.

Limitations

Single scaffold configuration. All rollouts use the same HUD MCP scaffold. Performance may differ under alternative tool configurations (e.g., Cursor IDE, custom sandboxes). A complete evaluation would require testing at least two scaffold configurations.

LLM-as-judge variance. The rubric is evaluated by claude-sonnet-4-6 with binary per-criterion decisions. Some criterion boundaries (e.g., "specific numeric values" for C1) involve judgment calls. We mitigate this with explicit tolerance bands in P2 and multi-path acceptance in P1, but residual inter-rater variance is not measured.

Domain specificity. This task tests ML training dynamics forensics specifically. Performance does not generalize to other domains. An agent that excels here may not perform well on software engineering, data pipeline, or general coding tasks.

Fixed artifact set. The environment stages a static snapshot of training artifacts. Real ML workflows involve iterative exploration. The air-gapped sandbox prevents agents from installing additional packages or accessing documentation, which may disadvantage agents that rely on external resources.

Token count as confound. Models with smaller context windows (gemini-2.5-pro) may struggle with the large staged data volumes. The 130 checkpoint directories and multi-MB parquet files require agents to be selective about what they read, favoring models with efficient exploration strategies over those that attempt exhaustive reads.

Citation

If you reference this task, its evaluation data, or the calibration results in publications, technical reports, or internal documentation, please cite:

@misc{jigsaw2026checkpoint, title = {Checkpoint Progression Forensics: An RL Environment for Training Dynamics Analysis}, author = {Jigsaw}, year = {2026}, url = {https://data.jigsaw.build/ml-engineering/checkpoint-progression-forensics/}, note = {Part of the Jigsaw ML Engineering evaluation suite} }

Calibration Profile