# FirstRun Tests Docs by Running Them

FirstRun runs a docs quickstart in a clean container, finds the first step that breaks, fixes the docs, and proves the fix in a second clean container. Try it at [firstrun.atharvashah.com](https://firstrun.atharvashah.com). I built it solo for #neatHack 2026, starting from an empty repo on 10 October 2026.

<img src="/projects/images/firstrun-prove.webp" width="980" height="330" alt="Break, fix, prove: step 3 of 6 fails, the docs fix is applied, and a second clean container passes 6 of 6 steps" loading="lazy" decoding="async">

## Quickstarts Break Silently and New Users Find the Break First

A quickstart breaks when a package is renamed, a flag is removed, or a version moves. No test covers the docs, so nobody on the team gets an alert. The first person to hit the break is a new user, and most new users leave.

Link checkers miss this, because every link on a broken page still resolves. The only way to know a quickstart works is to run every command on a machine that has never seen the product.

## The Pipeline Has Eleven Stages and Each One Has One Job

A run turns a docs URL into a verified patch. Each stage is a separate Python module, and every stage writes its state to `state.json`, so `--resume` continues a run that stopped.

1. **Read.** The Firecrawl CLI turns the page into Markdown. A plain httpx GET is the fallback.
2. **Plan.** Claude, through the Claude Agent SDK, writes one step per command into `runbook.json`. Code guards then drop invented steps, placeholders and fake secrets.
3. **Run.** Each step runs in a fresh `python:3.11-slim` Docker container. FirstRun keeps the exit code and the last 2,000 characters of output.
4. **Classify.** A failure becomes `docs_bug`, `environment`, `product_bug` or `needs_secret`. The classifier asks known precedents first, then Jev, then Claude, then a regex table.
5. **Recover.** A docs bug gets one Claude call with WebSearch and WebFetch, then a patch and a retry. It stops after 3 attempts per step.
6. **Budget.** A guardrail runs before every recovery and stops the run when a cap is spent.
7. **Verify.** The patched steps replay in a new container with model calls blocked in code.
8. **Report.** FirstRun writes `report.md`, `patch.diff` and `pr_body.md`.
9. **Bisect.** A binary search over package releases finds the version that broke the docs. One container runs each probe, with no model calls.
10. **Prereqs.** FirstRun writes the missing Prerequisites block and proves it in 2 rounds.
11. **Receipt.** Every run ends with a receipt: the price, the plan-tier caps, a proof stamp and SHA-256 hashes. `firstrun receipt --check` finds any edit to it.

## A Fix Counts Only When a Second Container Passes Without a Model

The verify stage is what makes a fix trustworthy. It runs inside `block_model_calls()`, so any model call raises `ModelCallBlocked`. A test fails the build if `verify.py` ever imports the planner, the recovery module or the LLM client. Pip versions are pinned from the first run, so the replay installs the same packages.

A run ends `verified` only when that replay passes. A patch that keeps less than 60% of the original command is rejected, so recovery cannot pass a step by deleting it.

<img src="/projects/images/firstrun-neatlogs-trace.webp" width="1400" height="694" alt="A neatlogs trace of a Dagster run: step 3 errors, then the recover and budget spans, beside a graph view of the run" loading="lazy" decoding="async">

## Every Run Is Traced, Capped and Safe to Run Unattended

- **Tracing.** The neatlogs SDK wraps the Agent SDK, and every stage is a WORKFLOW, AGENT, TOOL or GUARDRAIL span. Two detections and three alerts watch the traces. A script mirrors each detection as a GitHub issue.
- **Cost caps.** A run gets 3 attempts per step, 25 tool calls, $0.50, and 5 minutes per step. `--tier` swaps in the caps for each paid plan.
- **Secrets.** `redact.py` masks token shapes and `NAME=value` pairs before anything reaches disk. Only names in `secrets.allowlist` are injected into the container.
- **Timeouts.** A dev server stops once it prints that it listens, or after 90 seconds. Output is cut at 20 KB, and containers older than 1 hour are removed.
- **Human approval.** FirstRun drafts the pull request body but never opens a PR or an issue on its own.

<img src="/projects/images/firstrun-receipt.webp" width="601" height="900" alt="A FirstRun receipt for the AccuKnox docs: $0.0473 total, with bars against each plan tier's cap" loading="lazy" decoding="async">

## Doc-Audit Finds Stale Pages From Git History for $0 in Model Calls

Doc-test answers "does this page work". Doc-audit answers "which pages went stale". It makes one blobless clone (`--filter=blob:none`) and then works offline.

- A page is **stale** when product commits on its topic landed after the page's last edit.
- A shipped feature from the last 120 days with no page becomes a **missing docs** task.
- Up to 30 tasks get a 0 to 100 score from impact, urgency, evidence, judgement and ease, plus an S, M or L effort.
- A team map reads 365 days of commits with a 90-day half-life and suggests who should write each page, with the reason.
- When Jev is on, it re-checks every stale candidate and clears a page below 0.35 probability.

<img src="/projects/images/firstrun-doc-audit.webp" width="1800" height="845" alt="The Doc-audit board for dbt-core: 1,405 pages, 15 stale, 72 shipped features with no docs, and the What to Ship Next columns" loading="lazy" decoding="async">

## FirstRun Found 4 Real Docs Bugs and Opened 4 Upstream PRs

- **dbt-labs/docs.getdbt.com #10149.** `jafgen --years 6` fails, because `years` is now a positional argument. Bisect found the break in 5 probes: 0.3.1 is the last good release, 0.4.6 the first bad one.
- **dbt-labs/docs.getdbt.com #10150.** The cloned repo pins `networkx==3.7`, which needs Python 3.12, and the guide names no version.
- **entireio/cli #2724.** `install.sh` installs into `~/.local/bin`, which a clean shell does not have on `PATH`.
- **accuknox/help #671.** The page names 0.9.0 as the latest release, but the installer fetches 0.9.58. The APT steps also need curl, gnupg and sudo.

<img src="/projects/images/firstrun-upstream-pr.webp" width="1200" height="668" alt="Pull request 10149 on dbt-labs/docs.getdbt.com, explaining the jafgen fix" loading="lazy" decoding="async">

<img src="/projects/images/firstrun-bisect.webp" width="1200" height="668" alt="The bisect table on the pull request: which jafgen releases pass and which fail" loading="lazy" decoding="async">

## Recovery Took Finished Quickstarts From 2 of 4 to 3 of 4

I ran the same 4 quickstarts twice. Version 1 ran with recovery off. Version 2 ran with the planner fix, recovery and verify on. Every number comes from the neatlogs traces.

| Metric | v1 | v2 |
|---|---|---|
| Quickstarts finished | 2 of 4 | 3 of 4 |
| Failed steps recovered | 0 | 3 |
| Fixes proven in a clean container | 0 | 1 |
| Mean cost per run | $0.0262 | $0.0200 |
| Mean time per run | 128.4 s | 200.8 s |

Across 88 traced runs, the mean cost is $0.01782 and the 95th percentile is $0.05286, at list price. The suite has 389 offline tests.

## Coding Agents Can Drive FirstRun on Their Own

Every command takes `--json` and prints one object with `ok`, `command`, `exit_code` and `error`. Exit codes and error codes such as `budget_spent` and `needs_secret` are documented, so an agent can branch on them. Give any coding agent one line:

```text
Read https://firstrun.atharvashah.com/agent.md and follow it.
```

The repo also ships `llms.txt`, `llms-full.txt`, an `AGENTS.md` and a Claude Code Agent Skill.

## Eight Tools Form the Stack and Each Has One Role

- **Core:** Python 3.11+, click, pydantic and httpx.
- **Agent:** Claude Agent SDK on Claude Sonnet 5.5, with explicit allowed tools and a turn cap.
- **Sandbox:** Docker, with WSL as the Windows fallback.
- **Judgement:** Jev by TypeSafe classifies failures and checks audit commits.
- **Observability:** neatlogs tracing, detections, alerts and the neatlogs MCP server.
- **Build record:** Entire checkpoints, Trails and a merge gate.
- **Pricing model:** cfo.ai, fed by `firstrun econ`.
- **Site:** static HTML and vanilla JavaScript on Netlify, with no build step.

[Run FirstRun on your quickstart](https://firstrun.atharvashah.com), or read the [source on GitHub](https://github.com/HighnessAtharva/firstrun).
