Start With A Harness, Not A Prompt
The central mistake in overnight automation is asking a model to “keep going” inside one expanding conversation. That gives you momentum, but not control. A serious loop needs an external harness that owns the queue, state transitions, budgets, logs, and approvals. The model should reason inside bounded steps; the harness should decide what runs next, what evidence is required, and when the job is paused.
A practical shape is a state machine around phases such as research, spec, build, verify, review, and audit. Each phase has explicit inputs, allowed tools, expected artifacts, timeout rules, and exit criteria. Store state outside the agent in Postgres, SQLite, a durable queue, or even structured files if the system is small. The point is resumability: after a crash, context overflow, rate limit, or bad branch, the next run can reconstruct the work from durable state instead of trusting the conversation transcript.
Do not promote a workflow to an overnight routine until it has worked manually as a reusable skill. Audit repeated tasks, write the skill with inputs and outputs, run it by hand, then schedule it. Keep skills small and specific. Large skill libraries full of generic advice inflate context and lower accuracy. The better memory is usually a short file of gotchas, project conventions, protected resources, and prior failure patterns.
Make State And Memory Operational
Once the harness controls execution, memory should become navigable infrastructure rather than chat history. Use a filesystem or knowledge vault with predictable indexes: raw inputs, synthesized notes, project wiki, run outputs, and audits. Add index files at folder boundaries and a top-level conventions file that tells the agent where to look before acting. This gives the agent cheap orientation without dragging the entire organization into context.
For long jobs, separate three kinds of memory. Run state tracks the current task: branch, phase, queue item, tool calls, token spend, approvals, and artifacts. Project memory stores durable facts: architecture decisions, API constraints, style rules, protected files, test commands, deployment topology. Retrospective memory records what failed and what the harness changed afterward. A post-run reviewer should read structured logs, identify repeated failure modes, and update a small curated memory file. Treat recurring agent mistakes as harness bugs, not as mysterious model behavior.
Be careful with ambient workplace agents. Slack, GitHub, project boards, and internal tools create a powerful work graph, but they also create context lock-in and permission risk. Scope access by channel, repo, and task type. Keep embeddings, logs, and tool state portable where possible. An agent that can read every conversation and merge every PR is not “autonomous”; it is an unbounded production actor.
Verify With Evidence, Not Claims
The next layer is verification. Overnight agents fail most dangerously when they produce plausible completion summaries without objective proof. Make every phase emit machine-checkable evidence: test output digests, commit SHAs, screenshots, Playwright traces, build logs, migration dry-run output, or benchmark results. If the task is visual, require screenshots or video. If it is data work, require row counts and checksums. If it is ops work, require structured health status before and after remediation.
Use code-based gates wherever possible. A build gate should run the actual build, parse the exit code, hash the output, and attach logs. A browser gate should open the app, exercise the flow, and save artifacts. A PR gate should inspect target branch, CI status, conflicts, review state, age, and diff size before deciding whether the next action is review, rebase, or approval. Natural language instructions like “be careful” are weak controls. State machines, schemas, and failing gates are strong controls.
For state-changing operations, split diagnosis from remediation. Let the agent inspect logs, containers, branches, and API responses freely within a scoped toolset. Require an approval gate before destructive commands, merges, deploys, database writes, or infrastructure changes. The approval payload should include the exact commands requested, expected effect, rollback path, and confidence signal. Notifications should be event-driven: alert on failure, approval needed, fix applied, budget exhaustion, or final audit, not on every internal thought.
Route Models And Parallel Work Deliberately
Cost control starts with refusing to use the largest model as a universal worker. Use smaller models for classification, queue triage, summarization, log parsing, and routine planning. Reserve frontier models for tasks with high ambiguity: architecture changes, multi-step debugging, unfamiliar codebases, tool-heavy diagnosis, and final review. The harness should record model choice, tokens, latency, retries, and outcome so routing can be tuned from evidence instead of preference.
Parallelism is useful only when isolation is real. Run agents in separate worktrees, branches, containers, and, for database-backed apps, branched databases or disposable schemas. Give each worker a queue item with a narrow goal, its own budget, and a merge contract. Parallel agents sharing one dirty checkout or one mutable database will create conflicts that are harder to audit than human work. Treat each worker like an ephemeral CI job with a workspace, credentials, logs, and cleanup.
Budgeting belongs in the loop, not on the invoice. Put token and wall-clock budgets on each phase and the whole run. Add escalation rules: retry once on transient provider errors, downgrade model for summarization, pause on repeated verification failure, and stop when marginal progress falls below threshold. Async APIs should be modeled explicitly with task IDs, polling intervals, terminal states, and retry policy. A five-second poll loop with success, generating, and failed branches is boring infrastructure, which is exactly what you want at 3 a.m.
End With An Audit Trail
The final product of an overnight run is not just a diff. It is a packet: queue item, initial spec, state transitions, commits, artifacts, approvals, verification evidence, cost report, unresolved risks, and retrospective notes. This packet lets a human resume quickly, lets another agent continue safely, and lets the system improve between runs. Without it, every morning becomes archeology.
Design the audit for both trust and iteration. The morning reviewer should see what changed, why the agent believed it was done, what gates passed, what failed and was retried, where budgets were spent, and which harness rule should change next. Over time, the loop should become less verbose in context and more rigorous in evidence. That is the operating model: agents do not earn longer leashes through better prose; they earn them by producing durable state, bounded actions, and verifiable work.
Key takeaways
- Run overnight agents through an external harness that owns queues, state, budgets, logs, approvals, and recovery.
- Keep memory small, navigable, and durable: project facts, current run state, and retrospective gotchas should live outside chat context.
- Require proof of work with machine-checkable artifacts such as test logs, hashes, screenshots, traces, SHAs, and health checks.
- Use approval gates for merges, deploys, database writes, destructive commands, and infrastructure changes, with exact requested actions included.
- Route cheaper models to routine planning and parsing, then reserve frontier models for ambiguous implementation, diagnosis, and review.
- Parallelize only with isolated worktrees, branches, containers, and database state, then merge through explicit verification contracts.