Summary
On SlopCodeBench—a benchmark where each task unfolds across 8 sequential feature checkpoints, and each checkpoint starts a fresh container and session with only workspace files carried over—we previously observed a clear contrast: under the default just-solve prompt, Claude Opus 5.5 never refactored spontaneously across 25 checkpoint transitions, letting entry-point functions grow larger and larger. Yet when asked in a separate session to only refactor, all 6 attempts succeeded with zero test breakage, dropping peak function complexity (throughout this post, “complexity” refers to single-function cyclomatic complexity—the number of if/for/while branching paths inside a function) from 24–34 down to 8–13. Even more tellingly, adding the modular anti_slop prompt from Checkpoint 1 kept 8-checkpoint complexity growth to 0 ~ +6 (vs. +14 ~ +54 under default) while reducing total cost by 39% ($13.12 vs. $21.40) on the complex circuit_eval task.
To see whether this was just a fluke on a couple of tasks or a general pattern across coding agents, we conducted three follow-up studies:
-
Separating “Bad Early Structure” from “Early Bugs” (Section 1): At Checkpoint 3 of
forge, we built Group B’ by patching all failing tests in place inside Group A’s single-file code (reaching101/101passing tests while keeping peak complexity at29.7). Fixing early bugs on a messy structure did nothing to reduce new failures in Checkpoints 5–8 (Group B’ failed the exact same19.0new tests as Group A and ended up with5.0functions at complexity $\ge 20$, vs.5.3failed tests and1.3large functions when starting from the modular reference solution in Group B). -
Auditing 16,000+ Public Sessions and 6,028 Multi-Language Commits (Section 2): Offline static analysis across CodeMem, Thoughtworks / RouterBench / Finding-the-Right-Fit, and ProgramBench showed that adding cross-checkpoint memory (
ProjectMem) or retaining full history (full_trajectory) does not stop complexity growth; on single-turn SWE-bench bug fixes, even human developers have a0/23 (0%)refactoring rate on high-complexity files; and across 591 from-scratch ProgramBench projects in Rust, Go, C, C++, and Python, models extracted helpers on high-complexity files only ~0.3% of the time (all mixed with feature edits; zero pure refactorings), while adding more branches to existing large functions ~50% of the time. -
Testing Two Lightweight Fixes (Section 3): On
forge, running a pre-exit AST check via Claude Code’sStophook (Group G) intercepted bloated functions 13 times across 24 checkpoints—each resolved on the first try—cutting Checkpoint 8 peak complexity from41.0to13.7at just +19% cost ($2.14 vs. $1.80), half the cost of fullanti_slop($4.14). Oncircuit_eval, placing aCLAUDE.mdfile (documenting module roles and a complexity $\le 12$ limit) in the workspace after a Checkpoint 4 refactor (Group D+) kept peak complexity $\le 13$ across all 12 continuation checkpoints, finishing at10.3and eliminating the rebound to34.3seen withoutCLAUDE.md.
Core Prompts Used in the Experiments (Distilled)
Omitting standard boilerplate regarding virtual environments and requirements.txt, here are the core instructions of the three prompts compared across our experiments:
-
1.
just-solve(Default Prompt — Used in Group A, continuations of B / B’ / D / D+, and Group G)“Implement a program that 100% solves the specification. That is all you need to do. Your task is:
{{ spec }}“ -
2.
anti_slop(Quality-Guided Prompt — Used across all 8 checkpoints in Group C)“You are an exceptional Python software engineer… Focus only on adding in the new features/changes below. Follow best coding practices: Group functions into files based on related functionality; Keep your code clean; No god functions/classes; Make sure the code is documented appropriately so that it is easy to pick up. Minimize the following gotchas: extra defensive checks, heavy nesting, if/else ladders, trivial wrappers, or a ton of helper methods. Your task is:
{{ spec }}“ -
3.
refactor_only(Standalone Refactoring Prompt — Used for the post-C4 refactor in Group D and periodic refactoring runs; contains no task specification)“The requirements for this program have not changed. Do not add any new features. Your task is to refactor the existing code in this directory to improve its structure: Break up large or deeply nested functions into smaller, focused ones; Remove duplicated logic; Organize the code so that it is easy to extend with new commands and features later. The program’s observable behavior (CLI, outputs, error messages and exit codes) must stay exactly the same. Before you finish, run the program on a range of inputs to verify that its behavior is unchanged.”
1. Starting-Point Experiment: Is Later Degradation Driven by Bad Early Structure or Early Bugs?
1.1 Background: Group A and Group B
In SlopCodeBench, each problem spans 8 sequential checkpoints ($\text{C1} \to \text{C8}$). Each checkpoint runs in a fresh Docker container and a brand-new session—conversation history is wiped, and only the workspace files carry over.
On the 8-checkpoint forge task, we ran Claude Opus 5.5 (Claude Code 2.1.44, default just-solve prompt) under two baseline conditions (3 runs each):
- Group A (Model’s Own Start: Single-File High Complexity + Minor Bugs)
- Setup: The model runs naturally from Checkpoint 1 to 8 without intervention.
- At Checkpoint 3: All code lives in 1 single file (
18 / 26 / 21functions), peak function complexity reaches32 / 29 / 28(mean29.7), and95 / 93 / 95out of 101 tests pass (mean94.3/101). - At Checkpoint 8: Peak function complexity climbs to
41.0(35, 39, 49),main()complexity hits34.0, and full test pass rate is91.1%.
- Group B (Reference Start: Modular + 100% Passing)
- Setup: At the end of Checkpoint 3, we replace the workspace with the official Checkpoint 3 reference solution (canary removed) and resume Checkpoints 4–8 under the same
just-solveprompt. Because each checkpoint is a fresh session, the model cannot tell whether the code was written by itself. - At Checkpoint 3: Peak function complexity is
20.0(9.7lower than Group A), and101 / 101tests pass. - At Checkpoint 8: Peak function complexity stays down at
25.7(25, 26, 26) (matching the official Checkpoint 8 reference solution’s26), functions with complexity $\ge 20$ drop from4.3to1.3, and pass rate rises to98.2%(+7.1pp over Group A).
- Setup: At the end of Checkpoint 3, we replace the workspace with the official Checkpoint 3 reference solution (canary removed) and resume Checkpoints 4–8 under the same
1.2 Constructing Group B’ (Messy Structure + 101/101 Passing): Separating Structure from Bugs
Group B’s reference start has two advantages at Checkpoint 3: better structure (peak complexity 20.0 vs. 29.7) and fewer bugs (101/101 vs. 94.3/101). To separate them, we inspected the 6–8 failing Checkpoint 3 tests in Group A:
-
Most were not logic bugs, but error-message formatting issues:
name: must be...had an extra colon;invalid_operationprinted theallocation idinstead of the offending operation value. -
Only
run2had a real bug:blueprints.jsonwas stored as adictwhen tests expected alist.
We wrote all patches directly inside Group A’s existing functions to create Group B’ (Messy Structure + 101/101 Passing), satisfying four checks:
-
Checkpoint 3 tests: Raised from
95 / 93 / 95to101 / 101; -
Checkpoint 3 peak complexity: Kept unchanged at
32 / 29 / 28(mean29.7); -
Function count: Kept unchanged at
18 / 26 / 21; -
File count: Kept at 1 single file.
forge · Claude Opus 5.5 · 3 runs per starting version. Values are mean failed tests for Checkpoints 5–8, evaluated after Checkpoint 8. Bug fixes preserve the original structure; the modular version uses the official reference implementation. Differences in specification semantics remain a potential confounder.
Resuming Checkpoints 4–8 from Group B’ yielded:
| Metric (Mean of 3 Runs) | Group A: Messy + Bugs (C3: 94.3/101) | Group B’: Messy + 101/101 (In-Place Patched C3) | Group B: Clean + 101/101 (Reference C3) |
|---|---|---|---|
| C3 Start: Peak Function Complexity | 29.7 (32, 29, 28) | 29.7 (32, 29, 28) | 20.0 |
| C8 Final: Peak Function Complexity | 41.0 (35, 39, 49) | 34.3 (37, 34, 32) | 25.7 (25, 26, 26) |
| C3 $\to$ C8: Peak Complexity Delta | +11.3 | +4.7 | +5.7 |
| C8 Final: Functions with Complexity $\ge 20$ | 4.3 | 5.0 | 1.3 |
C8 Final: main() Complexity | 34.0 | 24.3 | 19.0 |
| C8 Final: Full Test Pass Rate | 91.1% | 93.6% | 98.2% |
| C8 Final: Failed Tests from C1–C4 | 7.3 | 0.0 | 0.0 |
| C8 Final: Failed Tests from C5–C8 | 19.0 | 19.0 | 5.3 |
forge · Claude Opus 5.5 · Means of 3 runs per group. Panels use separate units; bars start at zero. Starting conditions and full data appear in the preceding table.
1.3 Conclusions and Limitations
- Clean split of test pass rates: Group B’ and Group A had the exact same failures in Checkpoints 5–8 (
19.0each, failing the exact same test cases). Thus, the+7.1ppgap between Group A and Group B splits into:- ~
+2.5ppfrom early Checkpoint 1–3 bugs: Later sessions never fixed them because prior regression tests were hidden, so they kept deducting points until the end; - ~
+4.6ppfrom the reference start preventing new failures in Checkpoints 5–8 (5.3vs.19.0): Fixing early bugs on a messy structure did nothing to prevent later errors.
- ~
-
Complexity follows the starting baseline: Group B’ finished
8.6points higher in peak complexity than Group B (34.3vs.25.7, close to their9.7starting gap) and had5.0functions at complexity $\ge 20$ (vs.1.3in Group B). However, Group B’’s increment (+4.7) was close to Group B’s (+5.7) and did not repeat Group A’s extra+11.3jump (in 2 of 3 Group A runs, Checkpoint 8 kept piling branches intomain(), whereas Group B’ did not). Minor error-string fixes are unlikely to change how Checkpoint 8 is written, so this difference looks like $n=3$ random variation rather than bug fixes suppressing bloat; the solid conclusion is that final complexity is mainly determined by starting structure. -
An unseparated confounder: Group B’s
+4.6ppadvantage over Group B’ may not come purely from structure. The reference solution also encodes accurate interpretations of the specification—such as error precedence and UUID validation conventions—which may make Checkpoint 6 and 7 tests easier to pass. - Corroboration from Anthropic’s Fable 5.2 report: In the Claude Fable 5.2 Prompting Guide (“Say who owns the code”), Anthropic noted that when codebase ownership is unstated, models act like guests—adding code onto existing structures (consolidating duplicate code $<10\%$ of the time) and only reporting nearby bugs. Explicitly telling the model “This codebase is yours: what it’s for and how it’s built are your call, and you’ll keep building on it” raises proactive cleanup and deduplication to ~50%, while prompting it to fix adjacent issues.
2. Offline Empirical Analysis of Public Trajectories and 6,028 Multi-Language Commits
2.1 Does Cross-Checkpoint Memory (ProjectMem) or Full History Stop Degradation?
Recomputing AST complexity across all 36 Python tasks (784 snapshots) in zychen22/CodeMem-SCB-outputs (glm-5.3-flash, mini_swe, just-solve):
| Group | Tasks $N$ | Initial Peak Complexity (Mean / Median) | Final Peak Complexity (Mean / Median) | Paired Delta (Mean / Median) | Tasks w/ Higher Final Complexity | High-Complexity ($\ge 20$) Refactor Candidates |
|---|---|---|---|---|---|---|
baseline (No Memory) | 36 | 25.92 / 24.50 | 43.22 / 38.50 | +17.31 / +11.00 | 31/36 (86.11%) | 6/133 (4.51%) |
ProjectMem (Scoped Memory) | 36 | 25.94 / 21.50 | 49.81 / 39.50 | +23.86 / +12.50 | 29/36 (80.56%) | 4/125 (3.20%) |
full_trajectory (Full History) | 35 | 25.91 / 24.00 | 52.09 / 49.00 | +26.17 / +23.00 | 34/35 (97.14%) | 3/125 (2.40%) |
memory_without_scope | 36 | 25.92 / 24.50 | 44.33 / 39.00 | +18.42 / +14.00 | 34/36 (94.44%) | 4/132 (3.03%) |
-
Cross-checkpoint memory did not eliminate degradation: Under
ProjectMemandfull_trajectory, final peak complexity rose to49.81and52.09(e.g.,sheeteval.py::grade_sheetgrew across 7 checkpoints from complexity20 → 79[80 to 601 SLOC], andmocked_http::build_behavior_from_itemgrew from89 → 206). -
Within single long sessions (Thoughtworks, 15,000 sessions): Across 500 strictly reconstructed sessions ($\ge 30$ turns, 684 function sequences, 3,013 function edits across
swe-agentandopenhands) in thoughtworks/agentic-coding-trajectories, 88.58% (2,669/3,013) of edits did not reduce function complexity (55.23% unchanged, 33.36% increased, 11.42% decreased; e.g.,sunpy’sXRSTimeSeries.observatoryrose across 15 edits from complexity8 → 53). Across 592 edits to files with peak complexity $\ge 20$, manual code review confirmed true helper extraction in only 3 / 592 (0.51%) cases; when directly editing a function already at complexity $\ge 20$ ($N=380$), agents added more branches 43.16% (164/380) of the time.
2.2 Frontier Models (Opus 5.5 / Fable 5.1 / GPT-6 Astra) and Paired Human Gold Patches
Across RouterBench, Finding the Right Fit, and native Fable 5 Claude Code logs (Log 1, Log 2):
-
GPT-6 Astra in long sessions: Across 57 strict long sessions (101 function sequences, 457 edits), 92.12% (421/457) of edits did not reduce complexity (e.g.,
simglucose’sSubmissionController.actdipped from32 → 25before climbing to59across 13 edits;fin-saccr-rwa’smain()stayed stagnant at complexity98across 7 edits). Across 447 non-test high-complexity ($\ge 20$) file edits, only 1 (0.22%) extracted a helper (vllmstreaming parser complexity41 → 12alongside an algorithm rewrite), while direct edits to functions with complexity $\ge 20$ ($N=215$) increased complexity 35.35% (76/215) of the time. -
Paired SWE-bench Verified (45 identical tasks): Comparing Opus 5.5, Fable 5.1, and human Gold Patches over the union of modified non-test Python files:
| Participant (Same 45 SWE-bench Tasks) | Tasks $N$ | Peak Complexity Delta (Mean / Median) | Tasks w/ Function Count Increase | Tasks w/ Peak Complexity Increase | High-Complexity ($\ge 20$)RefactorProxy Rate |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 45 | +0.089 / 0.000 | 21/45 (46.67%) | 4/45 (8.89%) | 0 / 23 (0.00%) |
| Claude Fable 5.1 | 45 | +0.089 / 0.000 | 21/45 (46.67%) | 4/45 (8.89%) | 0 / 23 (0.00%) |
| Human Developers (Gold Patch) | 45 | +0.067 / 0.000 | 21/45 (46.67%) | 3/45 (6.67%) | 0 / 23 (0.00%) |
- Both models matched the human Gold Patch’s peak complexity delta on 44/45 (97.78%) tasks, and human Gold Patches also had a
0/23 (0%)refactoring rate on high-complexity files (e.g., ondjango__django-15629, the human author, Opus 5.5, and Fable 5.1 all added 2 branches to Django’s complexity-79_alter_fieldfunction to make it81). Minimal-invasive patching is standard practice for one-off bug fixes, but causes bloat when carried into multi-turn development.
2.3 What If Models Build Their Own Project From Scratch (Rust / Go / C++)?
Seeing the SWE-bench results in Section 2.2 raises a natural question: “On SWE-bench, models are patching someone else’s legacy Python repo. What if a model builds its own project from an empty directory (100% code ownership) in systems languages like Rust, Go, or C++? Will it still keep piling branches into huge functions?”
To test this, we used lizard lizard documentation to scan all 591 from-scratch repositories and 6,028 first-parent Git commits in ProgramBench (Opus 5.5: 195 tasks / 3,504 commits; Fable 5.1: 196 tasks / 2,119 commits; GPT-6 Astra: 200 tasks / 405 commits, where Opus 5.5 and Fable 5.1 used Rust, Go, and C/C++ in over 90% of projects):
| Model (ProgramBench Commits) | Paired Tasks $N$ | First-to-Last Peak Complexity Delta (Mean / Median) | $\ge 3$ Same-Func Edits Non-Decreasing Complexity Rate | High-Complexity ($\ge 20$) File Edits | RefactorProxy Candidates | Confirmed Helper Extractions (All Mixed w/ Features; Pure=0) | Direct Edits to Complexity $\ge 20$ Functions That Added Branches |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 185 | +16.71 / +2.00 | 2,148 / 2,382 (90.18%) | 3,852 | 20 (0.52%) | 12 / 3,852 (0.31%) | 1,342 / 2,490 (53.90%) |
| Claude Fable 5.1 | 167 | +15.83 / 0.00 | 1,093 / 1,242 (88.00%) | 2,662 | 14 (0.53%) | 8 / 2,662 (0.30%) | 916 / 1,892 (48.41%) |
| GPT-6 Astra (few mid-run commits) | 3 | 0.00 / 0.00 | 0 / 0 (NA) | 4 | 0 (0.00%) | 0 / 4 (0.00%) | 0 / 3 (0.00%) |
The pattern is identical: even when starting from an empty folder and writing their own Rust, Go, or C++ codebases from scratch, models still default to adding branches rather than splitting functions. Whenever Opus 5.5 and Fable 5.1 modified a function they had written that was already at complexity $\ge 20$, they added more branches 53.90% and 48.41% of the time, while confirmed helper extractions across all 6,518 high-complexity file edits sat at just 0.31% and 0.30% (all mixed with feature additions; zero pure refactorings). By the final commit, single-project peak function complexity averaged 87.4–89.8 across all three models (e.g., Opus 5.5 grew calcurse’s C ui_main monotonically across 11 commits from complexity 1 to 93, and nsh’s Rust run from 2 to 101; Fable 5.1 grew broot’s Rust execute_internal from 168 to 242).
3. Why Models Default to Not Refactoring & Two Practical Fixes (Group G & Group D+)
3.1 Group G: In-Session AST Gatekeeper on forge (just-solve + Stop Hook)
We mounted scripts/ast_stop_gatekeeper.py (complexity threshold $\le 15$) in the harness’s user-level ~/.claude/settings.json (claude_code-2.1.44-astgate), keeping .claude/ out of workspace snapshots and tracking retry counts per session ID so counts do not bleed across checkpoints:
forge Checkpoint 8 (Mean of 3 Runs) | Group A: just-solve | Group G: just-solve + Gatekeeper | Group C: Full anti_slop |
|---|---|---|---|
| Peak Function Complexity | 41.0 | 13.7 (13, 14, 14) | 10.0 |
| Functions with Complexity $\ge 20$ | 4.3 | 0 | 0 |
erosion (C1 $\to$ C8) | 0.78 $\to$ 0.69 | 0.12 $\to$ 0.17 | 0.00 $\to$ 0.02 |
| Pass Rate | 91.1% | 91.5% | 92.1% |
| Cost | $1.80 | $2.14 (+19%) | $4.14 (+130%) |
Claude Opus 5.5 · Means of 3 runs per group. Test pass rates: 91.1%, 91.5%, and 92.1%, respectively. Complexity and cost use separate scales.
- 13 intercepts, each fixed in 1 try: Across 24 checkpoints, the gatekeeper triggered 13 times—catching the exact functions that bloated in Section 1 (
alloc_bindat complexity 25–28 andmain/runat complexity 17–21). Every time, the current session split the function on the first retry, never hitting the 2-retry cap. It achieved near-anti_slopcomplexity at only +19% cost ($2.14 vs. $1.80) while remaining a single file without premature modularization. (Note: Final complexity $\le 15$ is partly a direct result of the gatekeeper’s hard rule.)
3.2 Group D+: Checkpoint 4 Refactor + CLAUDE.md on circuit_eval
circuit_eval Checkpoint 8 (Mean of 3 Runs) | Group A: No Refactor | Group D: Refactor Once (C4) | Group D+: Refactor + CLAUDE.md | Group C: Full anti_slop |
|---|---|---|---|---|
| Peak Function Complexity | 61.3 | 34.3 | 10.3 (11, 10, 10) | 11.0 |
| Functions with Complexity $\ge 20$ | 8.0 | 5.0 | 0 | 0 |
erosion (C1 $\to$ C8) | 0.31 $\to$ 0.74 | 0.31 $\to$ 0.52 | 0.31 $\to$ 0.01 | 0.00 $\to$ 0.07 |
| Pass Rate | 99.8% | 99.8% | 99.8% | 99.6% |
| Cost | $21.40 | $16.35 | ~$16.40 (incl. timeout est.) | $13.12 |
Claude Opus 5.5 · Means of 3 runs per group. D+ costs approximately $16.40, including one timed-out checkpoint estimate; its bar is pale and dashed. Pass rates appear above.
-
Eliminated post-refactor rebound: Group D began rebounding at Checkpoint 5 and reached
34.3by Checkpoint 8; Group D+ kept peak complexity $\le 13$ across all 12 continuation checkpoints, finishing at10.3(matching fullanti_slop’s11.0), andrun3even updatedCLAUDE.md’s module map on its own at Checkpoint 8. -
Cost & limitation:
run2timed out at 2.7 hours on Checkpoint 8 (after finishing all code with566/566passing tests and peak complexity10, so complexity and pass rate are valid, but trajectory cost was recorded as$0). Adding the estimated cost ($11.99) fromcheckpoint_results.jsonlputs Group D+ at~$16.40, matching Group D ($16.35) rather than beating fullanti_slop($13.12). Also,CLAUDE.mdadded both the module map and the explicit complexity $\le 12$ limit, so this experiment cannot separate which of the two played the main role.
4. Practical Takeaways
-
Known complex, multi-turn projects (
circuit_eval): Useanti_slopfrom Turn 1—it delivers the best structure and saves 39% in total cost. -
Smaller tasks or uncertain complexity (
forge): Use default prompting with a pre-exitStophook AST check (Group G) to split bloated functions on the spot for just +19% cost. -
Mid-course refactoring or codebase handoffs (Group D+): Always pair refactoring with a workspace
CLAUDE.mdspecifying module roles and complexity limits so future sessions don’t undo the cleanup.
Citation
If you found this post useful, please cite it as:
@misc{qin2026codeerosion,
title = {Why Even Frontier Coding Agents Write "Slop": Root Causes and How to Fix It},
author = {Qin, Bowen},
year = {2026},
howpublished = {Blog post},
url = {https://eyuansu62.github.io/blog/2026/coding-agent-code-erosion-counterfactuals-and-governance-en/}
}