Write tests, run them, fix failures

Preview: This loop runs across our internal repositories today. Its driver source lives in starlight-agent-config and is documented here before public standalone release. If you want it packaged first, open a request on GitHub issues.

Ask a coding agent to write tests for a module, run them, and fix failures, and it will try. What it will not do reliably on its own is stop after a bounded number of attempts, preserve an audit trail, or catch itself when it quietly weakens a test assertion to turn a red run green. The build-fix loop enforces those three invariants across every harness.

The six-step loopCopy Anchor Link

1. Take a baseline. Run the suite before touching code. Pre-existing failures belong to the baseline, not to your diff.
2. Reproduce first. For any bug fix, write the smallest test that fails for the reported reason and observe the failure before editing implementation code.
3. Fix the code, not the test. Modify a test only when the test itself is defective, and record the reason in the commit message.
4. Stop on the attempt budget. Five attempts by default, locked when the loop starts. If two consecutive runs fail with the same error signature, change approach instead of retrying.
5. Widen to the full suite. Once scoped tests pass, run the full test suite, typecheck, and linter. A scoped pass is not completion.
6. Cross-provider verification. A model from a different provider reviews the exact diff before the PR is marked ready.

What the driver verifies automaticallyCopy Anchor Link

• Multi-language runner detection: Discovers pnpm, npm, yarn, and bun scripts, node --test, pytest (via uv when present), unittest, cargo test, go test, and PowerShell Pester.
• Real execution proof: A zero exit code counts only when runner output proves at least one test executed. Skipped or todo-only runs are flagged as no-tests.
• Weakened-test guard: Snapshots test files and runner configurations (package.json, pytest.ini, tsconfig.json) before iteration 1, flagging deleted assertions, newly skipped tests, or narrowed test globs.
• Monotonic budget: Attempt counters decrement before each run begins so crashed runs count against the ceiling.