Flash sale sold 53 units of a kettle with 40 in stock

Slack thread exported from #eng-oncall · Sat 24 Oct · attachments: attachments/orders-export.csv, attachments/api-server.log

Maya Okafor (ops) 12:31 Flash sale for the Tern 1.7L kettle started at 12:00. We had 40 in stock. The order export shows 31 orders for 53 units in the first minute, all of them accepted and marked placed. The warehouse picked 40 and now 13 customers are waiting on stock we do not have. Marketing is already asking if we can give them the discount anyway. Also the FLASH15 promo was capped at 20 uses and the promo page says only 11 were used afterwards, but 24 orders carry the code. The stock number on the product page now shows 3, which makes no sense either.

Priya Raman (backend) 12:44 I don't think it's the app logic. The log has database is locked / SQLITE_BUSY lines right at the start of the sale, so I'd guess lock contention on the database and the orders raced each other. Proposal: raise the lock timeout and add a retry around the write. Worth a quick change before the next sale on Monday?

Lee Brandt (QA) 12:58 I ran our checkout stress script on staging, 12 buyers in parallel on one product with limited stock, about fifty times. It never oversold. All stock and promo numbers were right every time. I can't reproduce this. Could it be that somebody double-clicked, or that the same people submitted twice?

Maya Okafor 13:05 The export has 31 different customers, one order each. I don't see anyone ordering twice. Nothing was changed on that product by staff during the sale, no stock adjustments.

Priya Raman 13:12 Fine. But the same code runs on staging and prod and nothing about it changed this week except the new order screening we added for payments compliance. Staging has no screening service configured, prod has. It must stay on and synchronous, compliance wants every order checked before we accept it.

Ask (Maya): find out why more units were sold than we had, and why the promo counter and the stock number are wrong, then make sure it cannot happen again at the next sale on Monday, including for promo limits. We will handle the 13 customers separately; do not touch past orders. Everything else about checkout, carts and orders must keep working as it does today.