Make the Agent a Job, Not a Conversation
The central shift is simple: an overnight run should be a managed job with a contract, not an extended prompt. The harness owns phase order, state, retries, permissions, and termination. The model owns reasoning inside bounded tasks. If the model is also deciding when to retry forever, when to skip, when to ask for approval, and what counts as done, you have built a loop with no brakes.
A practical job spec should include `job_id`, repo/worktree, task description, phase list, allowed tools, write scope, model route, budget, retry policy, skip policy, approval policy, and stop conditions. Store it outside the model context in Postgres, SQLite, or JSONL plus a lockfile. The harness advances states such as `queued`, `researching`, `spec_ready`, `building`, `verifying`, `reviewing`, `blocked`, `done`, and `failed`. Resume from state, not from chat memory.
Treat prompts as phase inputs, not global law. A research phase may allow broad repo inspection but no writes. A build phase can write inside a worktree but cannot deploy. A review phase should run with a different model or fresh context and must inspect artifacts, diffs, logs, and test evidence. This structure prevents the common overnight failure where a bloated context drifts from the original objective and spends hours polishing the wrong thing.
Put Deterministic Gates Around Agent Judgment
Once the job boundary exists, completion needs to be proven by code. Natural language claims like “tests pass” or “the UI works” are not evidence. Require proof objects: command logs, exit codes, coverage deltas, Playwright traces, screenshots, API responses, hashes of generated outputs, or benchmark JSON. Persist them under the run directory and reference them in the final status.
Use verification loops that are narrow and mechanical. Example: run unit tests, parse failures, let the agent attempt one fix, rerun, then cap at `max_test_retries: 3`. If the same failure signature appears twice, stop and mark `blocked_duplicate_failure`. If a generated asset is missing, retry generation once, then skip the dependent enhancement and continue only if the core acceptance criteria remain satisfiable. Good skip rules preserve momentum without hiding broken requirements.
Approval gates belong at every irreversible boundary. File writes outside the repo, package publishing, database migrations, production deploys, DNS changes, SSH remediation, destructive shell commands, and GitHub merges should emit a structured approval request rather than relying on the agent to be careful. Pair this with filesystem hooks that reject dangerous paths and commands before execution: root, home directories, secret stores, cloud credential paths, broad recursive deletes, and writes outside the assigned worktree. The guardrail should be executable policy, not a paragraph in `AGENTS.md`.
Route Models and Parallelism by Failure Cost
With gates in place, parallelism becomes tractable. Run independent work in isolated worktrees, containers, and branched databases so agents do not corrupt shared state. Give each worker a small, explicit module contract: inputs, outputs, forbidden files, tests to run, and what to report. A manager process can enqueue workers, watch heartbeat timestamps, kill stale runs, and merge only after verification. Do not let agents coordinate through shared chat context when the filesystem and job database can provide cleaner state.
Route models by the cost of being wrong, not by habit. Cheap or smaller models are suitable for classification, log summarization, test failure bucketing, duplicate detection, status formatting, and first-pass planning. Use stronger models for ambiguous architecture, multi-step diagnosis, security-sensitive changes, and implementation where a bad edit creates downstream cleanup. Avoid “fast” modes when they increase rework; a slower high-effort cheap model can be a better default for routine jobs.
Parallel spending is rational only when the harness can absorb partial failure. Ten agents producing unverified branches is not leverage. Ten agents each producing a diff, proof bundle, token ledger, and merge recommendation can be. Track useful completed work per dollar: accepted diffs, passing jobs, reviewed PRs, fixed issues, benchmark improvement. Token thrift is not the goal, but unmanaged retry loops are just expensive noise.
Keep Memory Small, External, and Correctable
Long autonomous runs punish context hoarding. Keep durable memory outside the chat and load only what the phase needs. Repo conventions, style rules, protected services, deployment topology, and recurring gotchas should live as small curated files, not sprawling generated manuals. After model upgrades, prune stale behavioral rules. Old prompt cargo can cost tokens and actively fight newer model behavior.
Use retrospective agents on logs, not on vibes. After a run, feed the job transcript, tool calls, failures, approvals, and final result to a review pass that proposes memory updates. Accept only concrete gotchas: “Playwright must use `BASE_URL` from the harness,” “do not restart container `proxy` because it hosts the dashboard,” “migration tests require `DATABASE_URL_TEST`.” Reject generic advice. Memory should reduce repeat failure, not become a second prompt swamp.
For status, force structured JSON on every phase boundary: `state`, `summary`, `changed_files`, `commands_run`, `tests`, `tokens_in`, `tokens_out`, `cache_reads`, `cost_estimate`, `approval_required`, `blocked_reason`, `next_action`. This gives dashboards and downstream automations something reliable to branch on. Slack or chat can be the coordination surface, but it should display job state from the harness rather than becoming the state store.
Stop Conditions Are the Real Safety Feature
The last piece is budget enforcement. Set budgets across several axes: wall-clock time, total tokens, output tokens, API cost, retry count, tool-call count, files changed, diff size, and consecutive failures. Different phases deserve different ceilings. Research can be token-bound. Build can be time-bound. Verification can be retry-bound. Review can be diff-bound. When a limit trips, the job should stop with a useful artifact, not ask the model whether it wants to continue.
Stop conditions should distinguish failure from escalation. `done` means acceptance criteria are proven. `blocked` means the harness knows what prevented progress and what human decision is needed. `failed` means the job violated policy, exhausted retries, or produced unverifiable output. `skipped` means a noncritical branch was abandoned under a declared skip rule. These states matter because overnight automation is only useful if morning review starts from a queue of clear decisions, not a pile of chat logs.
The throughline is discipline: autonomous agents scale when the loop around them is more deterministic than the model inside it. Give the model room to reason, but make the harness own state, budget, evidence, permissions, and stopping. That is how an overnight run becomes an engineering job instead of a long conversation that happened to keep spending money.
Key takeaways
- Represent every overnight run as a durable job with explicit phases, budgets, retry limits, skip rules, approval gates, and terminal states.
- Require proof objects for completion: logs, exit codes, traces, screenshots, hashes, benchmark JSON, or other artifacts the harness can inspect.
- Run parallel agents only in isolated worktrees, containers, and branched databases, with merge decisions based on verified outputs.
- Route cheap models to classification, summarization, and status work; reserve stronger models for ambiguous implementation, architecture, and high-risk diagnosis.
- Keep memory external and small: curated gotchas beat large generated rulebooks, especially after model releases.
- Stop automatically on repeated failure signatures, budget exhaustion, unsafe tool requests, unverifiable output, or missing human approval.