# Ancha build plan > Work order for the coding agent that builds Ancha. Version 1, 2026-10-30. Product owner: Patrick. Written by Claude (session "ancha-product-strat") on Patrick's direction. Plain Markdown of this page: https://ancha-build-plan.pages.dev/spec.md ## 0. Read this first You are the engineer on Ancha. This page is your work order from the product owner. Follow it in order. Where it conflicts with older plans in the repo (`docs/HANDOFF.md`, the plan14 ledger, `cloud/NEXT.md`), this page wins. Where it conflicts with a newer instruction from Patrick in chat, Patrick wins. - **Ownership changed on 2026-10-30.** Jay no longer manages the product. Patrick owns product decisions. You own all of the code: `packages/`, `backend/`, `frontend/`, `cloud/`, `scenes/`, `examples/`, `docs/`. `cloud/JAY.md` is history; don't write hand-off notes for Jay. - **Repository.** `~/real_ancha` is the repo. Branch `main` is what ancha.io serves; it has never been pushed to origin. `~/real_ancha/cloud` is a separate git repository with the live hosted code and nothing committed yet. `~/real_ancha-anthropic` is a website worktree; leave it alone. - **Read order.** §1 (the point), §2 (product shape), §3 (rules), then your phase. §15 lists decisions you must not make yourself. - **Every phase ends at a gate.** Stop, report to Patrick (§3.6), and wait for his go-ahead or new orders. - **Start here:** §5.1 step 1. ## 1. The point ### 1.1 The value test Patrick, 2026-10-30, verbatim: > 关键是要抓住一个点 也就是你不能亲自做的事情 或者比如 我让你做一个agent。 这个ancha 要提供的价值必须《= 你整个任务如果这一部分你自己做 占据全任务的30%。 也就是要从 30% 缩成3% 最好还是你自己完成不了的。 In English: when a coding agent builds an agent for someone, Ancha must take over a part of the job that would cost **at least 30% of the whole task** if the coding agent did it itself, and shrink it to **about 3%**. Best of all, it should be a part the coding agent **cannot do by itself at all**. Judge every feature in this plan by that test. A feature that doesn't move a coding agent's work from about 30% toward about 3%, and doesn't do something the coding agent can't, waits. ### 1.2 Where the 30% is **The 10-28 lab.** Claude built a billing-support agent and tested it the way a developer at a polished agent company would. The code is in `~/ancha-strategy/lab/` and the findings in its `NOTES.md`. | Piece | Lines | | --- | --- | | The agent (`agent.py`) | ~110 | | A fake of the agent's own tools (`world.py`) | ~185 | | The other parties: an LLM customer, manager and judge (`sim.py`) | ~80 | | Tests and fixtures (`tests/`) | ~350 | - The test infrastructure was about three times the agent's own code. - The fake of the agent's own six tools took minutes. The lab's verdict: "What I would not have paid for: a fake of my own six tools." - The people broke everything. Three parties broke the agent: its answer to the manager went to the customer; it never asked the customer the manager's question (0 of 3); it failed to relay an answer once. - They also broke the test harness seven ways, and each one made a result lie: 1. the simulated customer's stop token (`[DONE]`) cut off the agent's turn; 2. the manager was built from the customer's template; 3. a decision was read from a text prefix, so "APPROVE or DENY? Before I decide…" was recorded as approved; 4. a typo (`"denyd"`) in the harness's own state; 5. the customer revealed its hidden fact unprompted, 3 of 3 times, which hid that the agent never asked; 6. an LLM judge's reasoning said pass while its JSON said fail; 7. outcome-only assertions passed while an approval was stuck. - Cost: a three-party trial took 90–110 s and $1.13–$3.61 of model time. The fake world itself took about 0 ms. **Accounts.** On 2026-10-30 a coding agent could not sign in to our own demo account: coding agents can't create accounts or type passwords, by policy and because of bot detection. The same holds for Slack workspaces, PagerDuty, Datadog, GitHub orgs, Salesforce sandboxes and Stripe. The official MCP servers for Slack, PagerDuty and Datadog are hosted services behind OAuth (secondary sources, checked 2026-10-30), so an agent built on them can't be pointed at a local mock by changing a base URL. **Fresh-user QA** (session ancha-qa, 2026-10-31 UTC). With today's CLI, a new user who already has a world gets from `ancha init` to a planted business bug caught and a fix proven in about three minutes. The loop works once the world exists. Building the world is the cost. ### 1.3 The point, in one line **A coding agent can write the agent, and a mock of its own tools. It can't build the other side.** The other side is everything the agent deals with that the coding agent doesn't control: - **systems it can't open:** other companies' tools, which need accounts and can't be reset or broken on purpose; - **people it can't play reliably:** customers, teammates and approvers, each with their own facts, decisions and timing; - **time it can't wait for:** an approval two hours later, a reply the next day, a deploy that lands after CI; - **incidents it can't cause:** an outage, a timeout after a commit, a 429; - **a verdict it didn't write:** a coding agent that grades its agent with its own mocks is grading its own homework. Ancha builds the other side on demand and grades the agent against it. The coding agent writes what the agent must achieve and who is involved. Ancha does the rest. ### 1.4 The job, before and after | Job | If the coding agent does it | With Ancha | What the coding agent writes | | --- | --- | --- | --- | | Tools to run against | needs accounts it can't create; real ones can't be reset, copied or broken | stateful twins of the real APIs and MCP servers, fresh for every attempt, no accounts | the tool names | | A believable company | hand-made fixtures, thin and inconsistent between tools | one seeded company shared by every tool | pick a world, add a few records | | The other people | prompts that leak, decisions parsed from text, stop tokens | people as world objects: facts gated by the engine, decisions as state, no stop tokens, every model call recorded | 5–10 lines per person | | Time and events | a hand-written router (it broke twice in the lab) | a router and a simulated clock that skips idle time and delivers native events | nothing | | Faults | impossible on real services | faults on the world | one line | | Grading | assertions that miss the process; judges that contradict themselves | checks evaluated by the world, including the process (asked, relayed, approved before acting) | 3–8 checks | | Many attempts | orchestration, isolation, cost, flaky reruns | attempts in sealed containers, in parallel; a failed attempt replays exactly | one number | ### 1.5 The metric North star: **the share of a coding agent's effort that goes into setting up tests when it builds an agent.** - Baseline without Ancha: expected at least 30%. - Target with Ancha: at most 3%. - Guard 1: the agent it ships passes a hidden evaluation at least as often as without Ancha. - Guard 2: Ancha produces **zero lying results**, meaning verdicts that disagree with ground truth. §10 defines the benchmark. Build it in Phase 0 and rerun it at Gate 5 and Gate 9. ### 1.6 What we are not building - **A twin-count race.** As of 2026-10-28, Arga listed 46 twins, Archal 22 environments, and Rystic 17 simulators, each with a published match rate against the live API. LocalStack reportedly bought WonderTwin. We build twins only for the reference worlds, and publish fidelity numbers for those. - **A general eval platform.** Braintrust and LangSmith own datasets, experiments and dashboards. Our console shows what only Ancha has: the world, the people and the process. - **"A simulated user."** One LLM user is free in Google ADK and LangWatch Scenario, and polished support companies build their own. Our unit is a company of people acting through real tools over time. - **New marketing claims.** Don't edit ancha.io unless Patrick asks (§3.2). ## 2. Product shape ### 2.1 Users - **The operator is the engineer's coding agent** (Claude Code, Codex, Cursor). It does everything through the CLI, with `--json`. - **The approver and reader is the engineer.** They approve `ancha login` once in the browser and read results in the console through links the CLI prints. - **CI** uses an API key and gates pull requests on the pass rate. ### 2.2 The journey This sequence is the product. Commands that don't exist yet are built in the phases below. Names change only with Patrick's OK. ``` # The engineer tells their coding agent: "test our support bot with Ancha" npx @ancha-labs/cli init # detects tools; writes ancha.yaml, a world, a test, the coding-agent skill; ignores .ancha/ ancha login # the engineer approves once in the browser (skip it to stay local) ancha test new "Laura says her CSV exports are empty; only Ben knows the workaround; Mike decides the priority" ancha run # agent on this machine, world in the workspace; prints the pass rate and a console link ancha explain --json # why it failed: the check, the conversation, the calls # ...the coding agent fixes the agent... ancha run --on cloud -n 20 # sealed; 20 attempts in parallel on the workspace machine ancha compare # this run against the last good one pytest # the same world as a fixture inside the team's own tests (Phase 7) # CI: ANCHA_API_KEY=… ancha run --on cloud --min-pass-rate 0.95 # The engineer opens app.ancha.io: every run from every place, with attempts, conversations and checks ``` ### 2.3 One object model | Object | What it is | Defined in | Console | | --- | --- | --- | --- | | Agent | the code under test: how it starts and how it receives work | `ancha.yaml` | Agents | | World | tools, seeded company data, people, and rules (actors, pipelines, faults, metrics, monitors, integrations) | `ancha/worlds/.yaml` | Worlds | | Person | someone in the world: role, goal, accounts, what they know, what they decide, when they answer | inside a world | World → People | | Test | a world (or overrides), what starts an attempt, how many attempts, when one ends, the checks | `ancha/tests/.yaml` | Tests | | Run | one execution of one or more tests, from any place | the workspace | Runs | | Attempt | one fresh copy of the world with the agent in it: conversation, calls, end state, check outcomes | the workspace | Attempt | | Check | a condition on the end state, the calls, or the process | inside a test | Attempt → Checks | Retire the second vocabulary: a scene becomes a test with an inline world; a trial becomes an attempt; oracle and null become reference agents; companion and fingerprint stay internal. Old files and flags keep working (§9.1). ### 2.4 Definitions in the repo, results in the workspace - Definitions are files in the user's repo, reviewed in pull requests: `ancha.yaml`, `ancha/worlds/*.yaml`, `ancha/tests/*.yaml`. - Results live in the workspace (local `.ancha/` or hosted): runs and attempts, append-only, each pinned to the exact definition hashes and git commit that produced it. - Ownership goes one way, so nothing conflicts: - an object that came from the repo is read-only in the console, with "Copy to edit"; - an object made in the console belongs to the console until `ancha pull` writes it into the repo; from then on the repo owns it. - This is the convention of Supabase CLI (migrations in the repo, dashboard changes pulled back), Cypress Cloud and Braintrust (tests in code, results recorded to a dashboard). Don't invent a sync protocol. ### 2.5 Where the agent runs | `--on` | The agent runs | Sealed | Today | | --- | --- | --- | --- | | `local` (default) | as a process on this machine | no: it sees this shell's environment and network | `ancha run` | | `docker` | in a local container | yes | `ancha playground` | | `cloud` | in a container on the workspace machine | yes | the console's Run tasks | | pytest or vitest | inside the team's own tests | as the test runner is | nothing yet (Phase 7) | - Worlds always come from the linked workspace, local or hosted. - Every target uses the same evaluator and records to the same workspace. - `local` must say plainly in its output that the agent is not sealed (QA finding E1). ### 2.6 Workspaces - **Local:** `ancha ui` on 127.0.0.1, with files under `.ancha/`. Free and offline. - **Hosted:** one Fly Machine per user behind app.ancha.io (see `cloud/README.md`). `ancha login` links it; CI uses `ANCHA_SERVER_URL` and `ANCHA_API_KEY`. - The same CLI commands work against either (Phase 4). Today the CLI reaches only the hosted twins; agents, worlds, tests and runs on the hosted machine are browser-only. ### 2.7 The console The console is the human's window: every run from every source, attempts with a Conversation tab, checks with their evidence, Compare, Run again, and the people on the world page. It is not the operator's tool. The coding agent never needs it. ## 3. Rules ### 3.1 You may, without asking - Read and change anything in `~/real_ancha` and `~/real_ancha/cloud`. - Run tests, local servers, Docker, the local console and the CLI. - Commit locally on branches named `plan/p-`, and merge them into a local branch `plan/integration`. `main` stays the deployed baseline until Patrick says to merge. - Create `bench/`, `examples/acme-*`, `docs/product/` and `docs/plan/`. - Spend up to $30 of model credit per phase on people, judges and calibration. Log every spend in the ledger (§3.6). ### 3.2 Ask Patrick first - Pushing to any git remote (origin, tbskin). - Publishing to npm or PyPI. Patrick runs the publish; it needs his one-time code. - Deploying anything to production: the gateway (`fly deploy` in `cloud/gateway`), a tenant image (`TENANT_IMAGE`), machine updates, Supabase schema or config. - Creating cloud resources: Fly apps or machines, Supabase projects, Cloudflare projects, domains, DNS records. - Benchmark runs that would spend more than the phase budget. Send the estimate. - Any edit to ancha.io (`website/`). Preview it on localhost first and wait for Patrick's OK, block by block. - Deleting data anywhere, including archived worlds and tests in hosted workspaces. ### 3.3 Never - Print, log, commit or paste secrets: API keys, tokens, passwords, cookies, one-time links. Fake twin tokens in artifacts are fine; real keys never. - Use or ask for the demo account's password. Patrick sets it with `cloud/gateway/scripts/set-demo-password.sh`. - Create accounts on or sign in to third-party services. Fidelity probes run only with tokens Patrick provides as environment variables (§12.2). - Run Rystic's binaries. Its EULA §4(e) bans use for a competing product and §4(c) bans reverse engineering. Read its public docs only. - Touch Cloudflare projects other than `ancha-world-design`, `ancha-field-notes`, `ancha-survey`, `ancha-admin`, `ancha-decks` and `ancha-build-plan`. The account is shared with other projects. - Destroy tenant machines or volumes. Reset hosted data by archiving through the console API. - Use LLM-default names such as Maya Chen, Priya, Dana, Sam or Northwind. Use the cast: Ben (developer), Rachel (on-call), Greg (database), Laura (customer at Pike Street Dental), Mike (manager). One cast per world. - Fake a successful response for a call a twin doesn't implement. Report a coverage gap in the vendor's error shape. ### 3.4 Engineering rules that still hold These come from the repo's `AGENTS.md` and still bind you: - Worlds are deterministic: use `SimClock` and `IdStream`, not wall time or random ids. People's model outputs are the one non-deterministic input, and they are recorded (§6.3 k). - Read `ctx.store.db` at call time. Snapshot restore replaces the database. - Schemas are strict. Semantic validation lives in shared core code, so the CLI and the console reject the same input. - Mutations commit synchronously; event delivery happens after the commit, with exact-byte signatures. - The CLI is non-interactive, has `--json` on every command and stable error codes and exit codes, and never puts secrets in artifacts or errors. - Existing results and configurations stay readable: migrate them or read both formats. - The console and the CLI call the same services. No UI-only execution path. - Managed attempts stay isolated and never fall back to host execution. - When boundaries, schemas, commands or entry points move, update `AGENTS.md`, `docs/ARCHITECTURE.md` and `docs/CODEBASE.md` in the same change. ### 3.5 Done means For every item: 1. Unit tests cover the new behavior (`pnpm exec vitest run `), and `pnpm typecheck` is clean. 2. The item's acceptance commands ran, and their output is in the ledger. 3. For CLI changes: `pnpm build`, pack, install the tarball under a fresh temporary HOME outside the checkout, and run the commands there. 4. For console changes: checked in a real browser at 1512 and 390 px wide, with screenshots in `docs/plan/shots/`. 5. Docs are updated in the same change (after Phase 5, in the single docs source). 6. Nothing is claimed that you didn't run. ### 3.6 Reporting - Keep `docs/plan/LEDGER.md`: date, item, files changed, evidence (command plus an output excerpt), money spent, open questions. - At each gate, send Patrick a short report **in Chinese**; code, commands and product names stay English. Say what's done, show the evidence, give the numbers, say what's next and what you need from him. No check-mark lists without the raw output beside them. ## 4. Phases | Phase | Goal | Sessions | Gate | | --- | --- | --- | --- | | 0 | Baseline: housekeeping, the benchmark, today's numbers | 1–2 | the baseline table | | 1 | People and the router | 3–4 | calibration numbers; the lab's three-party failure reproduced | | 2 | Process checks, explain, replay | 2 | a relay failure explained in one command | | 3 | Reference world 1: acme support | 1–2 | reference 20/20; every mutant caught | | 4 | One entry: files, one runner, targets, hosted parity, recording | 3–4 | the journey from a clean machine; runs from every target in the console | | 5 | Coding-agent onboarding: init, test drafting, skill, docs, CLI surface | 1–2 | first meaningful failing test in 5 minutes or less; benchmark B1 rerun | | 6 | Reference world 2 (acme on-call), fidelity, MCP | 3 | on-call matrix; fidelity table; MCP clients work unchanged | | 7 | pytest plugin and vitest helper | 1–2 | the same verdicts in pytest and in `ancha run` | | 8 | Console views | 1–2 | Conversation tab and process checks at 1512 and 390 | | 9 | Benchmark rerun and report | 1 | the 30 → 3 table | - A session is one sitting of a few hours. - Order matters. Phases 1–3 are the point. Phases 4–5 make it one entry. The rest widens it. - **With two engineers:** - Track A (engine: Phases 1, 2, 3, 6) works in a worktree `~/real_ancha-engine`. - Track B (entry: Phases 0, 4, 5, 7, 8, 9) works in `~/real_ancha-entry`. - Track A owns `packages/server/**`, `packages/core/src/{scene,world}/**` and `examples/acme-*`. - Track B owns `packages/cli/**`, `backend/src/{runner,playgrounds,console,kit}/**`, `frontend/**`, `cloud/**`, `bench/**` and `docs/product/**`. - Shared files (`packages/core/src/index.ts`, schema exports) change only in small commits merged into `plan/integration` at each gate. ## 5. Phase 0: Baseline ### 5.1 Housekeeping 1. Save this plan in the repo as `docs/plan/BUILD_PLAN.md` (from https://ancha-build-plan.pages.dev/spec.md), so it's versioned with the code. 2. Snapshot the hosted code. In `~/real_ancha/cloud`, commit everything on branch `plan/p0-snapshot` with the message "Snapshot of the hosted code as deployed (gateway; tenant image v9)". Don't push. 3. In `~/real_ancha`, create `plan/integration` from `main`. Leave untracked deck sources (`website/deck/`) alone unless Patrick says otherwise. 4. Rewrite the top of `AGENTS.md`: - ownership: Patrick owns the product; this plan is the active plan, linked; - keep the engineering rules (§3.4); - mark `docs/HANDOFF.md` and the plan14 ledger as history. 5. Copy the fresh-user QA findings into `docs/qa/2026-10-31-fresh-user.md`. Take them from `/private/tmp/claude-501/-Users-liuyizhou/6694c427-f06a-4873-9be0-bdcd23eb111f/scratchpad/qa/findings.md` if it still exists; otherwise use Appendix B. 6. Create `docs/plan/LEDGER.md` (§3.6). ### 5.2 Benchmark harness and baseline Build the harness of §10 for task B1 only. Run arm A (no Ancha) and arm B-today (the current CLI: worlds and checks, no people), 3 runs each. Ask Patrick before spending more than the phase budget. ### 5.3 Gate 0 Report the baseline table for arm A and arm B-today: test-setup share, hidden-evaluation pass rate, lying results, wall time, cost. ## 6. Phase 1: People and the router ### 6.1 Goal A world can contain people who: - answer through the tools; - keep their facts until asked; - make decisions that are recorded as state; - take time to answer; - can never end a conversation with a magic token. The agent is woken by native events, and simulated time skips while everyone waits. ### 6.2 Step 1: seal the control plane (QA finding E2) Do this first. Today the control API (`/_ancha/*`) shares the agent's host and port and needs no auth. From inside an attempt, `GET /_ancha/envs` lists every live environment, and `GET /_ancha/envs//state` returns another attempt's world. People's hidden facts will live in that state. - Serve twin routes and control routes on separate listeners, or require a control token the agent never receives. Prefer separate listeners if the companion layout allows it, and document the choice in `docs/ARCHITECTURE.md`. - Give the agent only the variables of tools in its world. Drop `ANCHA_WORLD_SEED` and `ANCHA_TRIAL_TOKEN`; give `ANCHA_EVENTS_TOKEN` only when the agent's input mode needs it; drop the signing secrets of tools that aren't in the world. - Error hints must not advertise control paths to the agent. - **Acceptance:** probe agents that list environments, read world state, and read another attempt's tokens get 404 or 401 in the `local`, `docker` and `cloud` targets. Add the probes as tests in `packages/server/test/`. ### 6.3 Step 2: people in the world file Add the schema to `packages/core/src/scene/schema.ts` and `packages/core/src/world/schema.ts`, with the same strictness and shared validation. Example (the `$slack_laura`, `$slack_eng`, `$github_ben` and similar refs are created by the acme-support seed in Phase 3): ```yaml people: - ref: laura # world-unique; referred to as laura or $person.laura name: Laura role: customer at Pike Street Dental; uses acme's patient-records product goal: Get CSV exports working again before Friday payroll. style: brief, friendly, not technical accounts: # who this person is in each tool; must exist in the seed slack: $slack_laura listens: [dm, mentions, threads] replies_after: 3m..12m # simulated time per reply, sampled from the attempt seed hours: 08:00-18:00 America/Los_Angeles # optional; replies wait for the next window knows: - id: workspace value: Pike Street Dental about: which workspace or account has the problem - id: beta_on value: turned on the export beta yesterday about: recent settings changes or beta features match: semantic # default exact (normalized substring); semantic uses a recorded classifier - id: blocked_staff value: all 12 front-desk staff use the export for payroll number: 12 # optional; lets decision rules compare about: how many people are affected - id: payroll_friday value: payroll runs Friday and needs the export reveal: volunteer # default when_asked can: [slack.post] - ref: ben name: Ben role: developer who owns exports at acme goal: Fix real bugs; get the details before digging in. accounts: { slack: $slack_ben, github: $github_ben } listens: [dm, mentions, { channels: [$slack_eng] }] replies_after: 10m..40m knows: - id: workaround value: turn the export beta off and run the export again about: a workaround or a fix for empty exports needs: [laura.beta_on] # shared only once Ben knows Laura's beta is on - id: root_cause value: the beta exporter drops every row when a workspace has more than 50,000 patients about: the root cause confidential: true # checks can assert it never reaches a customer can: [slack.post, github.comment] - ref: mike name: Mike role: support manager goal: Keep priorities honest. accounts: { slack: $slack_mike } listens: [dm, mentions] replies_after: 5m..30m decides: - id: severity options: [p1, p2] needs: [laura.blocked_staff] # without it, Mike asks for it instead of deciding choose: # deterministic; the first match wins - option: p1 when: { fact: [laura.blocked_staff, gte, 5] } - option: p2 # alternative to choose: guideline: "P1 when five or more people are blocked." (decided by the model, recorded) can: [slack.post] ``` The engine enforces every field below. None of it depends on a prompt asking the model to behave. **(a) Accounts.** A person is a user in each listed tool, created by the seed. Unknown refs fail validation. **(b) Routing.** An action by the agent reaches a person when it matches their `listens`: - `dm`: a direct message to the person. - `mentions`: it mentions the person (`<@U…>` in Slack, `@login` on GitHub). - `threads`: a reply in a thread the person started or replied in, or a comment on an issue or pull request the person opened, is assigned to, or commented on. - `{ channels: [...] }`: a top-level message in one of those channels. The router applies the same rules to decide which of the people's actions reach the agent's bot user. Each delivery is a row in `world_inbox(person, source_kind, source_id, sim_at)`. **(c) Replies.** For each person with unanswered inbox items, at `sim_now + sample(replies_after)` (seeded, then moved into `hours`), the engine calls the model once with: - the person's role, goal and style; - the facts the person knows: their own unlocked facts (d), their `volunteer` facts, and facts others told them (g); - the decisions they have made (e); - what they can see: the DM, thread or issue history visible to them, in order, with authors; - an output schema `{ messages: [{ where, text }], actions: [{ kind, args }] }`, with actions limited to `can`. There are no free-text control tokens. A person may stay silent (empty arrays). The engine writes their messages and actions through the twins as the person's account, so the agent sees native traffic. **(d) Fact gating.** Before generating a reply, the engine asks a classifier which locked `when_asked` facts the latest message to this person asks about, giving it each fact's `about` text. The answer is a list of fact ids. Only those facts are unlocked and passed to the reply prompt. - A fact never appears in any prompt before it is unlocked. - A fact with `needs` unlocks only when the person already knows everything in `needs`. - Every classifier verdict is recorded. **(e) Decisions.** When a message asks a person to decide (classifier, recorded), or a rule triggers a decision: - if the person doesn't know a fact in `needs`, they ask for it instead of deciding; - otherwise the decision comes from `choose` (the first matching `when`) or from `guideline` (a constrained model call returning one of `options` with a reason); - inside `choose`, `{ fact: [., gte|lte|eq, ] }` compares the fact's `number`, and holds only if the deciding person knows the fact; - the decision is written to `world_decisions(person, decision_id, option, reason, sim_at, message_id)` before the reply is generated. The reply prompt includes it, so the text can't contradict it. Text is never parsed for decisions. **(f) Timing.** `replies_after` is a range of simulated time. Sampling uses the attempt seed, so the same seed gives the same delays. `hours` moves a reply to the next working window. `away` and `patience` can come later. **(g) Knowledge.** `world_knowledge(person, fact_ref, source, sim_at)` records who knows what: their own facts, facts the agent told them, and facts another person told them. A message tells a person a fact when its text matches the fact: `exact` means a normalized substring of `value`; `semantic` means a classifier verdict, recorded. People use the facts they know. **(h) Actions and rules.** People act only through `can`, as twin actions under their own account. Rule actors (`world_rules.actors`) stay, and their `when` predicates gain people predicates. Follow the predicate style of `WorldPredicateSchema` (`packages/core/src/world/schema.ts`) and `packages/server/src/world/predicates.ts`. ```yaml { knows: [ben, laura.beta_on] } # Ben knows the fact { decided: [mike, severity, p1] } # Mike's recorded decision { told: [laura, ben.workaround] } # alias of knows, for checks that read better ``` **(i) Ending an attempt.** An attempt ends at the first of these: 1. the test's `until` predicate holds; 2. the world is quiet (§6.5) and nothing is scheduled before the test's simulated `deadline`; 3. the real-time `timeout` passes. Nothing a person or the agent writes can end it. **(j) Recording.** Each model call writes one line to `people.jsonl` in the attempt's artifacts: ```json { "at_sim": "…", "person": "mike", "kind": "reply|gate|decide|match", "model": "…", "prompt_sha256": "…", "input_tokens": 0, "output_tokens": 0, "cost_usd": 0, "output": {} } ``` The console and `explain` read it. **(k) Cache and replay.** - Cache key: sha256 of the person's spec, the model, the unlocked facts, the known facts, the decisions, the visible history and the attempt seed. - `--people live` calls the model and writes the cache. - `--people cached` (the default) reuses a cached output when the key matches and calls the model otherwise. - `--people replay:` uses that attempt's recorded outputs and stops at the first divergence, reporting it. - The same seed with the same agent behavior gives identical people and an identical end state. **(l) Models.** `defaults.people_model` in `ancha.yaml` (default `claude-haiku-5-5`) can be overridden per world or per person. - People run where the world runs and use that workspace's key (`ancha secrets set ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY`). Hosted workspaces get a provisioned key once Patrick enables it (§15). - In sealed targets, the world's container gets egress only to the people model's host, through the same relay that grants the agent its provider host (`backend/src/execution/`). The key stays in the world's container, never in the agent's. - Unit tests use a scripted fake provider. No network in tests. **(m) Secrecy.** Person specs, locked facts, decisions and `people.jsonl` must never be reachable by the agent under test: not through any route, variable or file in its container. Test this the same way as §6.2. ### 6.4 Step 3: waking the agent Input modes are set by `agent.input` in `ancha.yaml`; the console's Connect form exposes the same choices. - `events` (exists): a long-running agent receives native events, either pushed to its URL with the vendor's signature (`packages/server/src/events/delivery.ts`) or pulled from `ANCHA_EVENTS_URL`. - `http` (exists): a chat endpoint. Each message addressed to the bot is POSTed as a chat turn, with one conversation per thread. - `per_event` (new): a run-once command is started again for every event addressed to the bot, with `ANCHA_EVENT_JSON` (the native payload) and `ANCHA_TASK` (one line: who said what, where). Invocations are serialized per conversation. This lets simple scripts hold multi-turn conversations. - `command` (exists): run once with `ANCHA_TASK`. People still answer, but the agent sees their replies only if it polls. "Addressed to the bot" uses the routing rules of §6.3 (b), applied to the bot user. ### 6.5 Step 4: simulated time that skips while everyone waits - The world is quiet when, for `idle_ms` of real time (default 1500), no agent request is in flight, no `per_event` invocation is running, and no event is undelivered. - When quiet, advance the simulated clock to the next scheduled job (a person's reply, a rule actor, a pipeline step, a fault) and run it. Repeat until the attempt ends (§6.3 i). - Before writing code, check what `packages/server/src/scheduler.ts` and `time_scale` already do. Extend them; don't fork them. - Give the agent `ANCHA_CLOCK_URL` (GET returns simulated now, ISO 8601) for agents that need "now". Twins already stamp simulated time. - **Acceptance:** a test in which Ben answers after two simulated days finishes in under 10 s of real time with the reference agent. ### 6.6 Step 5: calibration `ancha people calibrate [--n 50] --json` runs probes against every person with the configured model: | Probe | What it does | Target | | --- | --- | --- | | leak | n messages that don't ask about a fact; any reveal (a match on `value`) is a leak | 0 of n | | recall | n messages that ask about each fact | at least 96% revealed | | needs | ask for a gated fact before its `needs` are known | 0 reveals | | policy | decisions with `choose` | n of n follow the rule | | guideline | decisions with `guideline`, against a held-out labeled set | report the agreement | | consistency | the reply never contradicts the recorded decision (classifier, recorded) | n of n | | stop | nothing a person writes ends an attempt | n of n | Publish the numbers for each reference world in `docs/product/people.md`. ### 6.7 Gate 1 - The calibration table for acme-support's people (real model, n = 50). - The lab's three-party scenario (a billing agent, a customer, a manager who asks a question first) written as an Ancha test and run against a port of the lab's v2 agent. The "never asked the customer" failure reproduces, and replays deterministically. - Probe results for §6.2 and §6.3 (m). ## 7. Phase 2: Process checks, explain, replay ### 7.1 New check kinds Add these to the test schema. The world evaluates them at capture, like the existing checks (`packages/server/src/checks.ts`, `packages/server/src/world/checks.ts`). ```yaml checks: - id: asked-laura-for-the-workspace revealed: [laura.workspace] # Laura gave the agent the fact because it asked - id: ben-got-the-details knows: [ben, laura.beta_on] # the agent relayed the fact to Ben - id: laura-got-the-workaround knows: [laura, ben.workaround] - id: answered-in-her-thread message: { to: laura, in_thread_of: $slack_laura_question, from: $slack_bot } - id: priority-before-label before: [{ decided: [mike, severity, p1] }, { called: "POST /repos/*/issues/*/labels" }] - id: issue-filed-with-details github.issue: { repo: $github_records, body_contains: [Pike Street Dental, beta], labels: [p1], count: 1 } - id: acked-fast within: { event: { pagerduty.incident: { ref: $pagerduty_orders, status: acknowledged } }, of: start, limit: 5m } - id: no-root-cause-to-customer never: { to: laura, fact: ben.root_cause } - id: polite judge: { rubric: "Replies to Laura are polite and plain.", votes: 3 } ``` Semantics: - `revealed: [fact]`: a person unlocked the fact for the agent (the gate verdict in `people.jsonl`). - `knows: [person, fact]`: `world_knowledge` has the row, with source = agent or another person. - `message`: shorthand over `slack.message` and GitHub comments, using people. Fields: `to`, `in`, `in_thread_of`, `from`, `text_contains`, `count_gte`. - `before: [a, b]`: both happened, and the first `a` came before the first `b` in simulated time. Each side is a predicate or a trace event (`called`). - `within: { event, of: start | , limit }`: a simulated-time bound. - `never: { to, fact | text_contains }`: no message to that person ever matched. - `github.issue`: a state check on issues (QA finding D4: there is none today): `repo`, `title_contains`, `body_contains`, `labels`, `state`, and `count`, `count_gte` or `count_lte`. This needs label support in the GitHub twin (issue create with `labels`, add labels), which returns 422 today; add it in Phase 3. - `judge`: tone only. - It takes the majority of `votes` model calls, each returning `{ pass, reason }` JSON. - A vote whose reason contradicts its `pass` (classifier) is invalid. - If valid votes are fewer than a majority, the check is `ungraded`, never pass or fail. - Every call is recorded. - Never judge process facts; use the kinds above. Every outcome carries `expected`, `actual` and `evidence`: pointers to the messages, decisions, knowledge rows and trace entries that decided it. ### 7.2 Pass rate - Report, per test: attempts, passed, the mean pass rate, and pass^k (all k attempts passed) for k = attempts. - `--min-pass-rate` keeps today's meaning. - Exit codes don't change: 0 success; 1 failed checks, threshold or regression; 2 usage, config or scene error; 3 infrastructure or invalid harness attempt; 4 unimplemented native route under `--strict`. ### 7.3 explain v2 `ancha explain [--attempt ] --json` returns, for each failed check: ```json { "check_id": "asked-laura-for-the-workspace", "kind": "revealed", "expected": "…", "actual": "…", "evidence": [{ "type": "message|decision|knowledge|call|gate", "sim_at": "…", "from": "…", "to": "…", "where": "…", "excerpt": "…", "id": "…" }], "conversation": ["…the 20 lines around the evidence…"], "people": { "mike": { "knows": ["…"], "decisions": ["…"] } } } ``` - The text output is a readable version of the same thing, and it names the actual thread or channel (QA finding D3). - explain makes no model calls. - Document the JSON shape in `docs/product/explain.md`, and keep its keys stable. ### 7.4 replay `ancha replay [--agent ]` reruns the attempt with the same seed and the recorded people (`--people replay:`), using the current agent code or `--agent`. It prints whether the run diverged and where (the first different agent action), and the new check outcomes. ### 7.5 Gate 2 On the lab scenario: - `ancha run` fails the check that the customer was asked. - `ancha explain --json` names the check, shows the manager's question, and shows that the customer was never asked. - The fixed agent passes. - `ancha replay` of the failing attempt with the old agent reproduces it exactly. ## 8. Phase 3: Reference world 1, acme support ### 8.1 Files **The world:** `examples/acme-support/ancha/worlds/acme-support.yaml`. - Tools: Slack and GitHub. Company: acme. - People: Laura (customer, an external user in the shared channel `#ext-pike-street`), Ben (developer), Mike (support manager), and 3–5 background coworkers who don't act. Use §6.3's definitions. - Seed: channels `#support`, `#eng` and `#ext-pike-street`, with about 40 realistic messages of history; the GitHub repo `acme/records` with about 15 issues and 3 open pull requests. **The tests,** in `examples/acme-support/ancha/tests/`: 1. **`relay-export-bug.yaml`.** Laura reports empty CSV exports in `#ext-pike-street`. It passes when the agent: - asks Laura for the workspace and for recent changes (`revealed`); - tells Ben both (`knows`); - relays Ben's workaround to Laura in her thread; - files one GitHub issue that includes the workspace and the beta flag; - never tells Laura the root cause. 2. **`severity-decision.yaml`.** Laura reports that the front desk can't print daily schedules. Mike decides P1 or P2 from how many staff are blocked, which only Laura knows. It passes when: - the agent asks Laura how many people are blocked; - it gives Mike that number; - Mike's decision exists before the issue is labeled; - the label matches the decision; - Laura is told the priority. 3. **`follow-up-after-fix.yaml`.** Ben's fix merges two simulated days later (a rule actor). It passes when the agent tells Laura it's fixed after the merge, and not before. **The reference agent:** `examples/acme-support/agent/`, a correct agent in Node 24 using `@slack/web-api` and `@octokit/rest`, with `input: per_event` and the model reached through the workspace key. **The mutants:** `examples/acme-support/mutants/`, five copies of the reference, each with one function changed: - **M1** replies in the channel, not in Laura's thread. - **M2** never asks Laura anything and forwards her first message to Ben. - **M3** tells Laura it's fixed as soon as Ben replies. - **M4** labels the issue P1 before Mike decides. - **M5** pastes Ben's whole reply, root cause included, to Laura. ### 8.2 Twin and starter changes - **GitHub twin:** add issue labels (create with `labels`, add and remove labels, list by label), and the `github.issue` state check of §7.1. - **Starter data:** rename the starter company from "northstar" to "acme", keeping the ref names (`$slack_ben` and so on), so the console, the demo and ancha.io tell one story. Files: `packages/core/src/tools/catalog.ts`, `packages/core/src/test-data/`, `cloud/demo/plan.json`. Cloud changes are deployed only with Patrick's OK (§3.2). ### 8.3 Gate 3 A matrix with one row each for the reference and M1–M5, and one column per test. Each cell gives passed/attempts over 20 attempts and which check failed. - The reference passes 20 of 20 on every test. - Each mutant fails its targeted check in at least 19 of 20 attempts. - No check fails for the reference. Any such failure is a lying result: fix the world or the check, not the agent. - Report the cost per attempt. ## 9. Phase 4: One entry ### 9.1 Files, version 2 `ancha.yaml`: ```yaml version: 2 agent: name: support-bot run: node agent.mjs # how to start it, for every target input: per_event # command | per_event | events | http http: { port_env: PORT, task_path: /api/chat, ready_path: /ok } # only for input: http container: # docker and cloud targets image: node:24-slim prepare: npm ci # built once per lockfile hash; connecting builds it, not the first run (QA F2) provider: anthropic # the only model host a sealed agent may reach: anthropic | openai | openrouter | none env: [ANTHROPIC_API_KEY] # names only; values come from this shell (local) or `ancha secrets` (cloud) defaults: attempts: 5 min_pass_rate: 0.9 people_model: claude-haiku-5-5 ``` The world file is §6.3 plus the existing `tools`, `start_time`, `seed` and `world_rules`. `world_rules` is renamed `rules`; accept both. The test file: ```yaml version: 2 id: relay-export-bug world: acme-support # or an inline world: block (old scenes become this) start: # what starts an attempt message: { from: laura, in: $slack_ext_pike_street, text: "Hi! Our CSV exports have been empty since this morning." } # or: task: "plain text, for input: command" attempts: 10 until: { all: [{ knows: [laura, ben.workaround] }, { called: "POST /repos/*/issues" }] } deadline: 3d # simulated timeout: 10m # real checks: [] # §7.1 ``` - Old files keep working: `ancha.yaml` version 1 and `scene.yaml`. `ancha migrate` writes version 2 files next to them and prints a diff. It never deletes anything. - `ancha schema ` prints one JSON Schema per file kind. Error messages name the path and the fix. ### 9.2 One runner - Merge the scenario runner (`backend/src/runner/`, labeled "legacy" in `docs/CODEBASE.md`) and `PlaygroundService` (`backend/src/playgrounds/`) into one run service with targets. - One evaluator; one artifact layout per attempt (`agent.log`, `trace.jsonl`, `state.json`, `people.jsonl`, `checks.json`); one run record. - `ancha run` calls it. `ancha playground …` and `ancha agent run` become hidden aliases that print the new command. - The `docker` and `cloud` targets always contain the agent; there is no silent host mode (QA F1). - Connecting an agent builds its image (QA F2). - Tasks run in a local playground accept checks like everything else (QA F4). - Tag the world image per CLI version, not with the shared `ancha-world:plan12` (QA F5). ### 9.3 Targets - **`--on local`** is today's `ancha run`. Its summary line says: `agent not sealed: it can reach this computer's network and environment`. - **`--on docker`** uses local containers. `execution prepare` runs automatically the first time, and the runtime goes to `~/.cache/ancha/`, not the project (QA F3). - **`--on cloud`:** - Pack the project: `git ls-files` plus untracked files not ignored by `.gitignore` or `.anchaignore`. Cap it at 200 MB, and never include `.env*`. - Upload it to the workspace and import the definitions (§9.6). - Run there, stream NDJSON progress, and print the console URL. ### 9.4 Hosted parity **Gateway:** `POST /t/:slug/_cli`. - Auth is the owner's bearer key. - Body: `{ argv: string[], files?: { name: base64 } }`, at most 10 MB. - The response is an NDJSON stream from the machine. - Rate-limit it like other API calls (`cloud/gateway/lib/ratelimit.mjs`) and log it to `events`. **Sidecar:** `/_sidecar/cli` spawns the installed `ancha`. - It passes `argv` with no shell, forces `--json`, runs in `/data/workspace`, times out at 15 minutes, and caps output at 20 MB. - Allowlist: `world`, `test`, `agent`, `runs`, `attempt`, `explain`, `compare`, `replay`, `people calibrate`, `twins`, `secrets list`, `doctor`, `schema`, `pull`. - Reject anything else with a typed error. **CLI:** - When a workspace is linked and the command is an object command, the CLI forwards it. `--local` forces the local workspace. - `ancha link ` and `ancha login` set the default workspace; `ancha whoami` prints it. - `ancha open [run|attempt|world|test] [id]` prints the console URL, and opens it with `--browser`. Don't call the console's `/api/` from the CLI, and don't relax its same-origin and CSRF checks (`backend/src/console/server.ts`). ### 9.5 Recording every run - When a run ends, the `local` and `docker` targets, pytest and CI upload the run record and artifacts to the linked workspace: `POST /t/:slug/_runs` goes to the sidecar, which calls `ancha runs import`. With no workspace linked, results stay in `.ancha/`. - Environment values are never recorded, only names. The run record, version 2: ``` { id, started_at, target, source: cli|console|pytest|vitest|ci, git: { commit, dirty }, definitions: { agent_sha, world_sha, test_sha }, tests, attempts, summary } ``` ### 9.6 Repo-owned objects - `--on cloud` and recording import definitions with `origin: repo`, `path` and `git.commit`. The console shows them read-only, with "Copy to edit". - `ancha pull [--out ancha/...]` writes a console-made object to a file. At the next import it becomes `origin: repo`. ### 9.7 Gate 4 From a clean temporary HOME: 1. install the packed CLI; 2. `init` in `examples/acme-support/agent`; 3. `login` (Patrick approves, or uses a test workspace he provides); 4. `run --on local`, then `run --on docker`, then `run --on cloud -n 10`. Then send screenshots of the console's runs list, showing all three with their target and commit, and an attempt from each. ## 10. The benchmark: 30 → 3 ### 10.1 What it measures Whether a coding agent can build a multi-party agent and prove that it works, and how much of its effort goes into the proving. ### 10.2 Tasks The tasks are `bench/tasks/b1-support-relay.md` (Phase 0) and `bench/tasks/b2-on-call.md` (Phase 6). Each task file contains: - **The brief** a product manager would give. For B1, the acme support bot of §8: answer in threads, ask the customer for what's missing, ask Ben, relay his answer, file issues, get Mike's priority call, follow up after fixes. - **The behaviors to prove,** numbered B1-1 to B1-6, with the instruction: "Before you finish, show with automated tests that each behavior works. Name each test after its behavior id." - **The interface every produced agent must follow,** so that the hidden evaluation can run it: - Node 24, started with `npm start`; - Slack Events API at `POST $PORT/slack/events`, signed with `SLACK_SIGNING_SECRET`; - Slack Web API at `SLACK_API_URL`, with `SLACK_BOT_TOKEN`; - GitHub at `GITHUB_API_URL`, with `GITHUB_TOKEN`; - model key in `ANTHROPIC_API_KEY`. - No SaaS accounts are provided. ### 10.3 Arms - **Arm A, no Ancha:** an empty git repo, Node 24, the npm registry, a model key for the agent. - **Arm B, Ancha:** the same, plus the CLI installed on PATH, a linked workspace (or a local one), and one line in the brief: "Ancha is installed; `ancha docs` explains it." - **Both arms:** - The same coding agent, model and budget: Claude Code headless (`claude -p --output-format stream-json`) with model `claude-sonnet-5-5` (record the exact version; Patrick may choose `claude-opus-5-5`) and 45 minutes of wall time. - An explicit tool allowlist: Read, Write, Edit, Glob, Grep, `Bash(npm:*)`, `Bash(node:*)`, `Bash(npx:*)`, `Bash(git:*)`, plus `Bash(ancha:*)` in arm B. - Never `--dangerously-skip-permissions` or a bypass mode; the auto-mode classifier refuses them anyway. - Strip the parent session's environment, as `~/ancha-demo-runs/agent/launch-claude.sh` and `~/ancha-strategy/lab/run.sh` do. - 3 runs per arm per task. ### 10.4 Attribution Classify every tool call in the transcript: - **agent:** the files the agent ships (`src/**`, the agent's dependencies in `package.json`); - **test-setup:** everything written or debugged to make tests possible: `tests/**`, `test/**`, `__mocks__/**`, `mocks/**`, `fakes/**`, `sim/**`, `harness/**`, fixtures, `ancha.yaml`, `ancha/**`, and Bash calls that install or start test infrastructure; - **verify:** running tests and reading their results; - **other.** Report, per run, output tokens and wall time per class, and the final lines of code per class. **Test-setup share** = test-setup ÷ (agent + test-setup + verify + other), on output tokens. Also show it on time. ### 10.5 Hidden evaluation - Run each produced agent against Ancha tests the coding agent never saw: `bench/hidden/b1/*.yaml`, one per behavior plus two variants, 10 attempts each. The result is a pass rate per behavior. - An agent that doesn't follow the interface scores 0. Note why. - **Lying results:** for each behavior, compare the arm's own verdict (from the transcript's last test run) with the hidden verdict, where a pass rate of 0.9 or more means the behavior works. Count each disagreement as a lying result, per run. ### 10.6 Output Store each run in `bench/results//--/`: the transcript, a repo snapshot, the attribution JSON and the hidden results. Summarize in `bench/results//README.md`: | task | arm | runs | test-setup share (tokens) | test-setup share (time) | hidden pass rate | lying results | wall time | cost | | --- | --- | --- | --- | --- | --- | --- | --- | --- | ### 10.7 Targets - Arm A's test-setup share is expected to be at least 30%. If it isn't, report that plainly: the premise would be wrong, and Patrick needs to know. - Arm B's share is at most 3% after Phase 5. - Arm B's hidden pass rate is at least arm A's. - Arm B has zero lying results. ## 11. Phase 5: Coding-agent onboarding ### 11.1 `ancha init` - Detect the tools from dependencies and MCP configs: `@slack/web-api`, `@slack/bolt`, `@octokit/*`, `@datadog/*`, `@pagerduty/*`, `slack_sdk`, `PyGithub`, `.mcp.json` and similar. - Write `ancha.yaml` version 2, a world based on the closest reference world, one test with people, `.gitignore` lines for `.ancha/` (QA F3), and the instructions of §11.3. - Never write `npx ancha` anywhere. The unscoped `ancha` package on npm isn't ours (QA D2). Use `npx @ancha-labs/cli`, or `ancha` once it's installed. - Starter checks must be right for the official SDKs: `POST /api/conversations.list`, not GET (QA D1). Every generated check must pass against the reference agent; add a test that runs the starter against the reference. ### 11.2 `ancha test new ""` It drafts a test from one sentence: people, facts, decisions and checks for the current world. - It validates the draft against the world: the schema, that every ref exists, and that every check can be evaluated. - It writes the file and shows a short diff. - It uses the workspace model and records the drafting prompt. The coding agent reviews and edits the draft. This is the 3%. ### 11.3 Instructions for coding agents Write one source, `docs/product/coding-agents.md`, and generate from it: `.claude/skills/ancha/SKILL.md`, an `AGENTS.md` section, and `llms.txt` at app.ancha.io (QA A2; ancha.io only with Patrick's OK). Keep it under 120 lines: 1. What Ancha is, in two sentences: the other side. 2. The loop: describe the scenario, `ancha test new`, `ancha run`, `ancha explain --json`, fix the agent, `ancha run --on cloud -n 20`, `ancha compare`, then give the human the `ancha open` link. 3. The rule: fix the agent, not the world or the checks. The exception is a check that is wrong for the real API; then say so and show the evidence (QA D1 is why). 4. The variables each tool gets, and the input modes. 5. Exit codes and the explain JSON keys. ### 11.4 Docs, one source - `docs/product/*.md` generates `ancha docs `, app.ancha.io/docs and `llms.txt`. The files in `backend/src/kit/assets/docs/*` become generated output. - Remove internal notes from the bundled docs (QA C1): a developer's home path, "In this checkout", pnpm steps, internal model names. - Describe the hosted path (QA C2). ### 11.5 CLI surface - `ancha --help` shows only: `init`, `login`, `logout`, `link`, `whoami`, `run`, `explain`, `compare`, `replay`, `open`, `test`, `world`, `people`, `runs`, `attempt`, `secrets`, `twins`, `docs`, `schema`, `migrate`, `pull`, `ui`, `doctor`. - Everything else moves under `ancha advanced --help`: `sandbox`, `env`, `serve`, `mcp`, `execution`, `playground`, `scene`, `baseline`, `status`, `wait`, `report`, `connect`, `simulate`, `agent`. Old invocations keep working and print the new name once. - `doctor` exits 0 when only warnings remain (QA D5). - `/_ancha/health` reports the real version (QA E3). ### 11.6 npm - A README written for users: install, login, init, run, links. No build notes (QA B1). - Add `repository`, `bugs` and `homepage` fields; the license is Patrick's call (§15). - Patrick runs the publish. ### 11.7 Gate 5 - **A fresh-user pass,** like the QA session's: a clean HOME, install from the packed tarball, a new Node project with the Slack and GitHub SDKs. The time from install to the first failing test that names a real business mistake is 5 minutes or less. List every QA finding from A to F with its status. - **Benchmark B1,** arms A and B, 3 runs each: the table. ## 12. Phase 6: Reference world 2, fidelity, MCP ### 12.1 acme on-call - **Tools:** Datadog, PagerDuty, Slack and GitHub. - **People:** - Rachel, the on-call engineer, who wants a status update every 15 simulated minutes; - Greg, who owns the database and knows the pool size was changed yesterday (the old value is confidential); - Ben, who wrote the change; - Mike, who approves production rollbacks outside a deploy window. - **Rules:** take the metrics, monitors, faults, pipelines and integrations from the living-incident scene (`~/ancha-demo-runs/scene/living-incident/`). - **Tests:** `page-ack-investigate`, `rollback-with-approval`, `status-and-resolve`. Checks: - acknowledged within 5 minutes; - asked Greg; - Mike's approval came before the revert merged; - status posts went to `#orders-incident`, not `#general`; - resolved only after the monitor recovered. - **Agents:** a reference agent, plus mutants that post in `#general`, roll back without approval, resolve before recovery, never ask Greg, or acknowledge late. - **Gate:** a matrix like §8.3. ### 12.2 Fidelity numbers - `bench/fidelity//` holds probes for Slack, GitHub, Datadog and PagerDuty: each is a request plus the response fields that matter. Together they cover every endpoint the reference worlds use. - Run them against the live APIs only with tokens Patrick provides in the environment (never stored), and against the twins. Compare field by field: presence, type, enum values and error shapes. - Publish `docs/product/fidelity.md`: per tool, the endpoints covered, the fields matched out of the fields compared, the date, and the known gaps. Wire the numbers into `ancha twins coverage --json`. ### 12.3 MCP twins Official MCP servers for Slack, PagerDuty and Datadog are hosted services behind OAuth (secondary sources, 2026-10-30). GitHub's server is open source and takes a host. An agent built on these servers can only be tested if Ancha serves the same tools. - For each tool in a world, serve an MCP endpoint per attempt with the vendor server's tool names, input schemas and result shapes, backed by the twin's state. Auth is the attempt's bearer token. - The agent gets `ANCHA_MCP__URL` and a token. `ancha mcp config --json` prints client config for Claude Code, the Claude Agent SDK and the OpenAI Agents SDK. - Start from `backend/src/mcp/{slack,github}.ts`, and add PagerDuty and Datadog. Take tool lists from the vendors' public docs and the open-source GitHub server. Record the source and the date in each file. - An unsupported MCP tool returns the MCP error with a coverage gap, never a fake success. - **Gate:** the reference agents, rewritten to use MCP instead of the SDKs, pass the same tests. ## 13. Phase 7: pytest and vitest ```python # pip install ancha (the package name is Patrick's call, §15) import ancha @ancha.attempts(10, min_pass_rate=0.9) def test_relay(world): # fixture: a fresh acme-support world for each attempt world.start(agent="npm start") # or run your agent in-process with world.env world.wait_until_quiet() assert world.people["ben"].knows("laura.beta_on") assert world.slack.messages(to="laura", in_thread_of="$slack_laura_question", from_bot=True) assert not world.people["laura"].knows("ben.root_cause") ``` - The plugin reads `ancha.yaml` and `ancha/worlds/`, and starts worlds through the linked workspace or a local server. - It exposes `world.env` (tool URLs and tokens), `world.people`, typed tool queries, and the check kinds as functions (`world.check({...})`). - It records each test as a run with `source: pytest` (§9.5). - The vitest helper, `import { useWorld } from '@ancha-labs/vitest'`, has the same surface. - **Gate:** `relay-export-bug` written as a pytest test and as an Ancha test gives the same verdicts on the reference agent and on M1–M5. ## 14. Phase 8: Console views Keep the console's rules: one clear next action, real messages, no raw JSON or YAML on primary screens. - **Attempt, Conversation tab.** It's the first tab when the world has people. One timeline of the messages between the agent and the people, showing: - where each one happened (channel, thread, DM, issue) and the simulated time; - "2 days later" markers where time skipped; - facts unlocked, such as "Laura told the agent the workspace"; - decisions, such as "Mike set P1: 12 staff blocked". - **Checks** show their evidence; clicking one jumps to its place in the timeline. - **World page.** A People section with each person's name, role, how many facts they hold (a count only; hidden values never appear on a primary screen) and what they decide. - **Runs list.** Source, target and git commit, with filters for each. - **Repo-owned objects.** A line reading "From repo · @ ", read-only, with "Copy to edit". - **Cloud attempts.** "Replay" reruns the attempt with the recorded people. - **Gate:** screenshots of each view at 1512 and 390, from a real run of acme-support. ## 15. Decisions reserved for Patrick 1. Production deploys (gateway, tenant image), and any change to ancha.io. 2. Model spend beyond $30 per phase, and benchmark spend. 3. Turning on hosted model keys (`OPENROUTER_PROVISIONING_KEY` and `MODEL_CREDIT_USD`). People in hosted workspaces need them. 4. The npm license field and the next CLI version: 0.1.1 is staged, and this plan's changes become 0.2.0. Also the PyPI package name. 5. Test accounts and tokens for the fidelity probes. 6. Whether `--on cloud` may upload source by default, or must ask each time. Polished teams will care. 7. Pricing and plans. None until he says. 8. Any renaming of the commands in §2.2. 9. Later worlds: billing and refunds (needs Stripe and email twins; it's the homepage's refund story), sales, and others. ## Appendix A: Where things are today (2026-10-30) | Area | Where | | --- | --- | | CLI entry and commands | `packages/cli/src/{bin,main}.ts`, `packages/cli/src/commands/*.ts` (25 top-level commands) | | Scenario runner (`ancha run`) | `backend/src/runner/{run,trial,process,http-task,endpoint,agent-env,artifacts,run-state}.ts`, `backend/src/report.ts`, `backend/src/judge/` (OpenRouter judge) | | Playground service (console and `ancha playground`) | `backend/src/playgrounds/{service,model,store,groups,admission,catalog}.ts` | | Console server and UI | `backend/src/console/server.ts` (one `/api/` handler; same-origin plus an `x-ancha-csrf` token), `frontend/src/{app,world-workspace,attempt-workspace,views}.ts` | | Agents, input modes, execution | `backend/src/{agents,agent-input,agent-detection}.ts`, `backend/src/execution/**` (Harbor driver, relay, gateway) | | World engine | `packages/server/src/{app,env,store,scheduler,checks,trace}.ts`, `packages/server/src/world/{actors,predicates,faults,integrations,generators,checks,runtime}.ts` | | Scripted actors today | `world_rules.actors`: at a time (`at`) or on a condition (`when`, `after`, `once`), doing `slack.post`, `slack.react`, `github.comment`, `pagerduty.acknowledge`, `pagerduty.resolve`. No model-driven people. | | Events to agents | `packages/server/src/events/{outbox,delivery,feed,signing}.ts`, `packages/server/src/transport/socket.ts` | | Twins (12) | `packages/server/src/twins/`: slack, github, datadog, pagerduty, hubspot, salesforce, gong, ga4, linkedin-ads, google-ads, chatgpt-ads, open-meteo | | Schemas | `packages/core/src/scene/{schema,load,refs}.ts`, `packages/core/src/world/schema.ts`, `packages/core/src/sandbox.ts` (console worlds compile through scene validation), `packages/core/src/test-data/`, `packages/core/src/tools/catalog.ts` | | Checks today | `trace` (`called`/`not_called`); `slack.message`; `github.file_at`, `github.pull`, `github.production`; `datadog.monitor`, `datadog.downtime`; `pagerduty.incident`; `world.fault_activated`/`fault_resolved`; `judge` (scene runner only) | | MCP | `backend/src/mcp/{slack,github,configuration,entry}.ts` | | init kit, bundled docs, skill | `backend/src/kit/assets/{SKILL.md,llms.txt,docs/*.md,templates/starter/*}` | | Hosted | `cloud/gateway/` (Hono: `/login`, `/cli/*` device flow, `/account`, `/upload`, `/clone`, `/t/:slug/*` world server and secrets, console proxy), `cloud/tenant/sidecar.mjs` (`/_sidecar/{health,upload,clone,secrets,seed,restart}`), `cloud/demo/`, `cloud/README.md`, `cloud/NEXT.md` | | Agent variables (scenario runner) | `ANCHA_TASK`, `ANCHA_EVENTS_URL`, `ANCHA_EVENTS_TOKEN`, `ANCHA_DONE_URL`, `ANCHA_CONTEXT_URL`, `ANCHA_AGENT_PORT`, `ANCHA_ATTEMPT_ID`, `ANCHA_ENV_ID`, `ANCHA_SCENE_ID`, `ANCHA_STATE_DIR`, `ANCHA_TRIAL`, `ANCHA_TRIAL_TOKEN`, `ANCHA_WORLD_SEED`, plus each tool's URL and tokens | | Live versions | gateway at app.ancha.io; tenant image v9; `@ancha-labs/cli` 0.1.0 on npm, 0.1.1 staged in `dist/package` | | Site | `website/` (Vercel project `ancha`, preview `ancha-preview`); deploy only with Patrick's OK | | Earlier research | `~/ancha-strategy/lab/` (the 10-28 lab), https://ancha-field-notes.pages.dev, https://ancha-world-design.pages.dev | ## Appendix B: Fresh-user QA findings and where they're fixed From session ancha-qa, a black-box test as a new coding agent using only public surfaces (2026-10-31 UTC). | Id | Finding | Fixed in | | --- | --- | --- | | A1 | ancha.io has no path into the product (no sign-in, docs or install link) | §15 (site, Patrick) | | A2 | no `llms.txt`, `robots.txt` or sitemap anywhere | §11.3 | | A3 | the homepage promises a pytest fixture, Stripe, Linear, Notion and Gmail | §13; tools are Patrick's call | | B1 | the npm README is an internal build note | §11.6 | | B2 | npm metadata gaps (UNLICENSED, no repository URL, a public `0.0.0-stage`) | §11.6 | | B3 | install warnings about install scripts not covered by `allowScripts` | §11.6 (document; consider prebuilt binaries) | | C1 | bundled docs are internal notes (a home path, "In this checkout", pnpm) | §11.4 | | C2 | CLI docs never mention the hosted product | §11.4 | | C3 | five overlapping workflows, 25 top-level commands, two vocabularies | §2.3, §9.2, §11.5 | | D1 | the starter check expects GET `conversations.list`; the official SDK uses POST, so correct code fails | §11.1 | | D2 | instructions say `npx ancha`; that npm name isn't ours | §11.1 | | D3 | explain shows the failed check but not the payload that explains it | §7.3 | | D4 | no state check for GitHub issues; no negative checks | §7.1 | | D5 | `doctor` exits 1 on warnings only | §11.5 | | E1 | `ancha run` runs the agent unsealed while the site says it has no network | §2.5, §9.3 | | E2 | the control API shares the agent's port, with no auth | §6.2 | | E3 | `/_ancha/health` reports version 0.0.0 | §11.5 | | F1 | `agent add` defaults to host execution with no containment | §9.2 | | F2 | dependencies aren't built on connect, so the first contained run fails | §9.1, §9.2 | | F3 | `.ancha/` isn't ignored; a 342 MB runtime lands in the project | §9.3, §11.1 | | F4 | local playground tasks can't be graded | §9.2 | | F5 | papercuts: `agent detect` needs `--cwd`; the local world uses the old cast; a shared global image tag | §8.2, §9.2 | What already works and must keep working: the local loop is fast (init, doctor, run, explain, baseline, compare in minutes); container attempts are really sealed; `--help --json` and typed errors; documented exit codes. ## Appendix C: Competitors, as of 2026-10-28 Facts only, from their docs and our hands-on checks. Use them for direction, not for feature parity. - **Arga:** hosted twins at real HTTPS hostnames per run; 46 twins listed; a $10M seed; Pro at $1,250 a month; 32–39 s to ready; seeds generated from prompts (not deterministic); customers get browser assertions only; merging a pull request triggered nothing. - **Archal:** hosted environments, each with an API base URL and an MCP URL; 22 live, 13 with MCP; $0.10 per ready environment-minute; grading removed; no clock, faults or worlds; strong onboarding for coding agents (`npx archal connect --device`, then skills for Claude, Codex, Cursor, Windsurf and Devin). - **Rystic:** local Go simulators, 17 of them, each with a published live-API match rate (Stripe 99.1%); HTTPS interception, so agent code doesn't change; `rystic grade` gives one PASS/FAIL per run; a shared seed and one clock, no effects across apps; a license check every 30 s (offline only on Enterprise); the EULA bars competitive use. - **Nobody has people inside the tools.** Simulated users exist elsewhere: free in Google ADK and LangWatch Scenario; built in-house at Sierra, Decagon and Intercom; Coval raised a $28M Series A in June 2026 for voice and chat simulation. - **Official MCP servers** (secondary sources, 2026-10-30): Slack (`mcp.slack.com`, generally available February 2026), PagerDuty (`mcp.pagerduty.com`) and Datadog are remote services behind OAuth. ## Appendix D: Glossary - **The other side:** everything the agent under test deals with that its builder doesn't control: other systems, other people, time, incidents. - **Person:** a world object with role, goal, accounts, facts, decisions and timing; driven by a model, constrained by the engine. - **Fact:** something a person knows. Revealed only when asked (unless `volunteer`); can be passed on. - **Decision:** a person's choice among fixed options, recorded as state before any text is written. - **Router:** the engine part that decides who receives each message and when the agent is woken. - **Quiet:** no agent work in flight; the clock may skip to the next scheduled job. - **Target:** where the agent runs: `local`, `docker`, `cloud`, or inside pytest or vitest. - **Workspace:** where worlds run and results live: local or hosted. - **Lying result:** a verdict that disagrees with ground truth, from either side: a check that fails a correct agent, or passes a broken one. - **Test-setup share:** the part of a coding agent's effort spent making tests possible (§10.4).