Rails 24:00 Orvane Home & Outdoor

Accuracy against the answer key

The world was generated with every planted fault written to an answer key. This screen scores the ledger against it: Sat 21 Nov 18:00 to Fri 27 Nov 23:50, the week before the sale and the sale evening, replayed on a copy of the world.

90%precision: detections that were real faults
97%recall: planted faults it found
83%handled without a person
9actions rolled back by people
4 daysmean time to resolve, from the fault's start (return abuse included)
Weakest detector: Address not on file, precision 73% on 22 detections. It is the first place a person should look, and the first to tune.
incident kinddetectionsprecisionplantedrecalltime to resolve
Address not on file22 73%
2080%
1.9 min
Carrier delay cluster7 100%
7100%
4.5 h
Chargeback deadline12 100%
12100%
5.9 h
Duplicate order53 94%
50100%
0.9 min
Late-shipment risk10 100%
5100%
53 min
Suppressed listing17 100%
17100%
32 min
Oversell31 87%
27100%
22 min
Price below cost47 83%
39100%
2.0 min
Return abuse35 91%
3494%
24 days
Review bomb2 100%
2100%
26 min

Decided by

236 incidents by rules. Rules decide wherever the written policy leaves no choice; the model is asked only where there is a real one.

The sale evening alone

18:00 to midnight: precision 88% on 97 detections, handled without a person 92%, 8 rolled back, weakest Price below cost at 64%. Evening recall counts only faults that started in those six hours. From out/replay-seed23.json.