Rails architecture, in full
This is the long version of Rails in ten minutes: every part, what it owns, how the parts talk, and the choices behind them. Paths are relative to the repository root. Every figure quoted comes from the seed 23 hold-out run described in the README, with the command next to it there.
1. Overview
Rails has six parts and one rule: the part that thinks never holds the authority to act.
| part | package | owns |
|---|---|---|
| world | rails/world | the synthetic retailer, generated from a seed, and its answer key |
| dictionary | rails/dictionary | the definitions: SQL, owner, version, parameters, tests, change history |
| operator | rails/agent | detectors, cadence, runbooks, rules, the optional model, Controller, tools, scripted on-call people, replay |
| ledger | rails/ledger | the append-only record, approvals, rejections, rollbacks, repeats |
| accuracy | rails/recal | matching detections to the answer key; precision, recall, time to resolve, drift |
| screens | app | the rails, play mode, ledger, approvals, accuracy, definitions, Ask, these documents |
All state lives in one DuckDB file. The world tables, the ledger, the detector run times and the replay markers sit side by side, so a replay is a file copy and an audit is a SQL query.
2. The world (rails/world)
python -m rails.world.generate --seed N --out data/rails.duckdb --stats writes a complete retailer in about ten seconds, identical for a given seed. Seed 7 is used for development and tests, seed 11 for a second tuning pass, seed 23 is held out.
- Catalogue. 40,000 products in categories with a long tail of best sellers, unit costs and list prices, and delivery lots per product (a review bomb follows one faulty lot, not the whole product).
- Channels. The own shop and two marketplaces, Marketvine and Tradewick, each with its share of orders, the share of the catalogue it lists, a markup, a fee, a ship-by window (48, 16 and 36 hours) and a chargeback answer window. Seed 23 has 89,943 listings, each with its own price, shown quantity and last sync time.
- Fulfilment. Three warehouses (North hub, Central hub, South hub) with stock per product and an hourly pick capacity; 18 lanes from warehouse to region, each with a carrier and a backup carrier; carriers collect at fixed times of day, so shipments leave in batches.
- Demand. 170,000 customers with addresses; 303,155 orders over 90 days on daily and weekly curves, ending at the midnight after the sale day. The sale evening (Friday 18:00 to 24:00) takes 12,162 orders against about 1,500 in the same hours of a normal Friday. Orders have lines, payments, picks, shipments, deliveries, cancellations and carrier scans.
- After the sale. Returns with reasons and conditions, chargebacks with deadlines, reviews per product and lot, customer messages, promotions.
- The answer key.
incidents_truthholds every planted fault: kind, start and end, subject, and the correct handling. Seed 23 has 414 rows across the ten kinds. The generator also plants look-alikes that should not be acted on: approved loss leaders below cost, honest customers who change an address once, slow lanes that are slow every day, returns that are high but honest.
Times are naive local time in the world's one time zone. docs/world.md describes the tables and the planted kinds in detail.
3. The dictionary (rails/dictionary)
Each term is a YAML file in rails/dictionary/definitions/: a title, a description, parameters with defaults, and one or more variants. A variant has a key, an owner, a version, a change note, its SQL and its tests. Detectors are terms too: the SQL of oversell is the oversell detector.
- One source of truth. The detectors, the operator, the screens and Ask call
load()and evaluate terms by name. A number on the Ask screen is computed by the same SQL the operator acts on. - Two meanings, kept apart.
late_shipmenthas a marketplace variant (handed to the carrier after the ship-by time) and a customer variant (delivered after the promised date).active_listinghas a marketplace variant (on sale) and a merchandising variant (on sale with stock to sell).clean_returnhas a warehouse variant (came back resaleable) and a trust variant (resaleable, for a fit or preference reason, from an account below the abuse line).stock_on_handhas a canonical variant (available: on hand minus reserved) and a physical one. When a term has two meanings and no canonical one, Ask refuses to pick: it answers with both and counts the overlap by key. - Tests on the world. Each variant carries tests (row counts within bounds, a column never null, one meaning a subset of another). The Definitions screen runs them on the current world on request.
- Change history. Parameter changes append to a history file with who and why; the replay records definition changes as markers so a shift in a detector's numbers can be traced to the change that caused it.
- Tuning record.
rails/dictionary/TUNING.mdlists every detector change made while tuning on seeds 7 and 11, with the numbers before and after. Nothing was changed after the seed 23 run.
4. The operator (rails/agent)
4.1 The pass
runner.step() is one pass at the clock's current logical time:
- housekeeping (a hook for timed states that end by themselves; no Rails action sets one, so it is empty),
- the detectors that are due by cadence (
cadence.py: realtime every pass, hourly, daily at 08:00; last runs are kept indetector_runsso a restart does not repeat a daily check), - items held back by the business-wide rate limit,
- new incidents, deduplicated against the 15-minute window and against incidents a person already rejected or rolled back (a person said no: the operator leaves that incident alone),
- proposals, in parallel, then decisions and actions one at a time in severity order.
Proposals may run on worker threads because they only think; decisions, tool calls and ledger writes stay on the runner's thread, in order, because budgets and cooldowns depend on what was just done.
4.2 Runbooks, rules and the model
policy.yaml gives each kind a runbook: the actions the operator may consider, in the order it tries them. A rule per kind (rules.py) fills in the arguments from the evidence. For oversell, duplicate orders and chargebacks the policy says rules decide. Elsewhere, when more than one action is possible, the optional model (llm.py, any OpenAI-compatible endpoint, configured only through the environment) is asked to choose and explain. A model answer that is not valid JSON, names an action outside the runbook, or misses an argument is discarded and the rule decides; the ledger records which happened. Proposals for the same kind, subject and evidence are reused within a run, so a repeated sighting never costs a second call. The seed 23 numbers in the README were produced with rules only.
4.3 The Controller
controller.py is the only code that can authorise an action. For every proposal it checks, in order:
- the action is known and not on the
neverlist (unknown means tier 3), - the action is in this kind's runbook,
- the arguments are complete, and the targets they name appear in the incident's evidence,
- the action's written condition holds (an evidence pack needs a delivery proof; a refund above EUR 100 needs a person),
- the action is outside its cooldown on this subject,
- budget: per subject per hour (4 automatic actions on one listing, order, lane or warehouse), per action per hour (30 listing pauses, 200 rerouted orders), and business-wide per minute (40 realtime actions; back-office kinds have their own limit and never use the realtime budget).
The outcome follows when_a_limit_is_hit: a used-up subject budget goes to a person, the business-wide limit waits for the next minute, a cooldown or a false condition is declined with the reason logged, an unknown action is never taken. Budgets are counted from the ledger, so they survive restarts and can be audited with a query.
4.4 Tools
Sixteen tools in rails/agent/tools/, each named after the call it stands for: resync_stock (PUT /listings/{id}/inventory), file_evidence_pack (POST /disputes/{id}/evidence), reroute_fulfilment (POST /fulfilment/orders/reassign), hold_order, resubmit_listing, update_tracking_eta, pause_listing, switch_lane_carrier, quarantine_batch, flag_account_review, request_carrier_intercept, refund_order, open_ticket, restore_price, block_account, accept_chargeback. Each one reads the rows it will touch, applies the change in the world tables, and returns the before and after snapshot with the call string. Undo is generic: it plans the compensating updates from the snapshot and applies them as new events, so history is never edited. A runbook step can chain a follow-up (hold_order then refund_order for a held duplicate); the follow-up goes through the Controller like any other action, so a refund above the limit still waits for a person.
4.5 Notifications and stakes
Tier 1 actions and tier 2 asks write a draft to the outbox for the people the policy names (the category manager, the logistics lead, the trading lead on call). Nothing is sent by Rails. stakes.py attaches what is at stake to every row, from the incident's own evidence: the money already hit or due and the orders affected. It never claims money "saved".
4.6 Scripted on-call people
The world is synthetic, so the people in it are too. humans.yaml defines a small shift (a trading lead on call, a trust analyst, payments, a logistics lead) with hours and response delays, and rules for how they answer: approve a duplicate refund, reject blocking an account whose returns are mostly resaleable, roll back a paused listing that was an approved loss leader. Their decisions land after their delay on the logical clock, as normal ledger approvals, rejections and rollbacks, each labelled "scripted on-call responses (synthetic)". Anything no rule covers stays in the queue for a person at the screen.
4.7 Process gaps
A kind with no runbook gives the operator no authority. It files one backlog row per kind in process_gaps and, when people have handled similar cases through the queue, drafts a runbook from what they approved, rejected and reverted. A person turns the draft into policy; the operator never does.
4.8 Shadow mode
With RAILS_SHADOW=1 the Controller decides as usual, but tier 0 and tier 1 actions are written as shadow rows saying what would have been done, and no tool runs. Tier 2 items still go to the queue.
5. The ledger (rails/ledger)
One table, ledger, append-only. Each row has: logical time, incident id, kind, subject, the proposal (source: rules or model, rationale, the model's own words when there is one), the Controller's decision and reason, the action and arguments, the tier, the result (call string, before and after, the undo function), the evidence, the definition term and version, model name and token counts, status, the person, a note, and the parent row.
Statuses: auto_done, shadow, escalated (waiting for a person), approved, rejected, rolled_back, declined, failed. Approving or rejecting an escalated row, or rolling back an executed one, writes a child row pointing at it; the original row is never changed. The approval queue is the set of escalated rows still without a decision. Repeats of the same problem inside the dedupe window go to ledger_repeats and are shown on the original row.
6. Replay and accuracy (rails/agent/replay.py, rails/recal)
scripts/replay_sale_evening.py refuses to run on data/ and expects a copy of the world. It replays the days before the sale at 5-minute steps (rules only, fast), then the sale evening at 30-second steps, with the scripted shift answering. Each part is marked in replay_events. With --save-start it saves a consistent copy of the database at the moment the sale evening opens: the start snapshot for live play. The run writes out/replay-<label>.json with the ledger summary, step timings and the accuracy figures.
Seed 23 wall times on a laptop: 70 to 80 seconds for six days of history (1,728 steps), 52 to 55 seconds for the evening (720 steps, 72 to 77 ms per step on average, under 190 ms at the slowest).
rails/recal/matching.py matches each detection to the answer key by kind, subject and time, with 30 minutes of slack either side. metrics.py computes precision (detections that match a planted fault), recall (planted faults matched), the share of incidents resolved by the operator's own action, reverted actions, time to resolve per kind (from the fault's start as its detector saw it to the resolving action), and which incidents rules or a model decided. drift.py watches the model's confidence, invalid answers and reverts over time. The Accuracy screen computes all of it from the ledger in the database it is pointed at, and names the weakest detector.
7. The screens (app)
A FastAPI application with Jinja templates and one small script, started with .venv/bin/uvicorn app.main:app --port 8167. RAILS_UI_SERVICES=real (the only mode) reads the database named by RAILS_DB, and the server refuses to start without it.
7.1 The rails
The main screen draws one frame of the evening at a logical moment, from /rails.json:
- Stations. For each of placed, paid, picked, shipped, delivered and returned: the orders sitting there now, per channel, and the arrivals in the last 15 minutes. An order is on the line from the moment it is placed until it is delivered; orders delivered or returned during the evening stay at those stations. Cancelled orders leave the line.
- Dots. Every order whose id is divisible by 23 is drawn as a dot on its channel's track, between its station and the next, further along the longer it has waited (scaled to how long orders usually wait at that station, the 90th percentile in the world). Because the sample is fixed, an order keeps its dot from frame to frame and glides to the next station when it moves.
- Side rails. One per incident kind seen so far, along the evening's time axis. Each incident the operator opened is a dot at the time it acted, coloured by its outcome as of now (done, waiting for a person, rolled back, declined), with a ring for tier 2. The newest one on each side rail carries its receipt. In play, a new incident leaves the main line at the station it came from and slides down onto its side rail.
- Warehouses. For each warehouse: units free to sell against units on hand, the change since 18:00, SKUs at zero, and orders paid but not picked against the hourly pick capacity. Stock at a past moment is the stock at the world's end plus what was picked after that moment.
- Receipts. The latest five actions in plain words, each linked to its ledger row.
?at=HH:MM shows a fixed moment of the recorded run without touching the play clock.
7.2 Play mode
app/play.py keeps a logical clock over the evening that advances lazily at 10, 60 or 240 times wall speed. app/live.py runs the operator live: "From 18:00" copies the start snapshot to a working file, and a background thread with its own database connection steps the real operator on it (detectors, Controller, tools, ledger and the scripted shift) one 30-second tick at a time, as far as the play clock has gone. Every screen reads the working file while it runs, so approving an ask on the Approvals screen during play acts on the live run. The play clock is held to at most a minute ahead of the operator. "Recorded run to midnight" stops the live run and returns every screen to the replayed database. Without a start snapshot, play walks through the recorded run instead. On seed 23 a full live run of the evening writes the same 129 ledger rows by kind and status as the recorded run, in about 45 seconds of wall time (scripts/live_vs_recorded.py).
7.3 The other screens
Ledger (filters by kind, outcome and tier; each row opens with its evidence, chain, before-and-after and a rollback button where an undo exists), Approvals (the tier 2 queue with approve and reject, and the latest decisions by people), Accuracy, Definitions (with the tests runnable on the current world), Ask, and these documents rendered from docs/. A visitor's site arm (v, p query parameters) rides on every link, form and redirect.
8. Choices and their reasons
- Logical time everywhere. The world, the operator, the scripted people and the screens share one logical clock, so a replay is exact, a live run at 240 times speed is the same code as a run at 1 times, and accuracy can be measured against a key with times in it.
- Rules by default, a model where it adds something. Most incidents in a retailer's evening have one right action. A model on those adds cost and variance and nothing else. Where there is a real choice, the model proposes and the Controller still decides.
- Authority in a file people sign. The policy is readable by a head of operations, versioned, and enforced in code that the model cannot reach.
- Append-only record. Approvals, rejections and rollbacks are new rows. The ledger is the audit trail, the budget counter and the accuracy input at once.
- Undo by compensation. Tools snapshot before and after and undo with a new change, so the record of what happened is never rewritten.
9. Plugging in a real retailer
- Map the order hub, channel connector, warehouse system, carrier feed and payments desk onto the tables in
rails/world/schema.sql, through read-only replicas or change feeds. - Run the dictionary's tests on the mapped data; fix the mappings until they pass.
- Run in shadow mode for a few weeks and score the "would do" rows against what people actually did.
- Point each tool at the real call it already names, with credentials scoped to exactly that call.
- Hand over tier 0 kind by kind, starting with the detectors whose precision held up in shadow mode.