Accuracy against the answer key
The world was generated with every planted fault written to an answer key. This screen scores the ledger against it: Sat 21 Nov 18:00 to Fri 27 Nov 23:50, the week before the sale and the sale evening, replayed on a copy of the world.
90%precision: detections that were real faults
97%recall: planted faults it found
83%handled without a person
9actions rolled back by people
4 daysmean time to resolve, from the fault's start (return abuse included)
Weakest detector: Address not on file, precision 73% on 22 detections. It is the first place a person should look, and the first to tune.
| incident kind | detections | precision | planted | recall | time to resolve | ||
|---|---|---|---|---|---|---|---|
| Address not on file | 22 | 73% | 20 | 80% | 1.9 min | ||
| Carrier delay cluster | 7 | 100% | 7 | 100% | 4.5 h | ||
| Chargeback deadline | 12 | 100% | 12 | 100% | 5.9 h | ||
| Duplicate order | 53 | 94% | 50 | 100% | 0.9 min | ||
| Late-shipment risk | 10 | 100% | 5 | 100% | 53 min | ||
| Suppressed listing | 17 | 100% | 17 | 100% | 32 min | ||
| Oversell | 31 | 87% | 27 | 100% | 22 min | ||
| Price below cost | 47 | 83% | 39 | 100% | 2.0 min | ||
| Return abuse | 35 | 91% | 34 | 94% | 24 days | ||
| Review bomb | 2 | 100% | 2 | 100% | 26 min |
Decided by
236 incidents by rules. Rules decide wherever the written policy leaves no choice; the model is asked only where there is a real one.
The sale evening alone
18:00 to midnight: precision 88% on 97 detections, handled without a person 92%, 8 rolled back, weakest Price below cost at 64%. Evening recall counts only faults that started in those six hours. From out/replay-seed23.json.