Balance sweeps between an organization's accounts fail at the top of the hour

Component: Transfers (support's guess) · Linked: PAY-2380 (PostgreSQL 17 upgrade, done 9 Oct)

From: Marta Lindqvist, head of engineering, Northwind Coffee Roasters · Received: Thu 22 Oct, 8:40 am (Stockholm)

Since the middle of last week our balance sweeps between our three Paylane accounts (the roastery, Wholesale and the cafés) fail several times a day with your generic "something went wrong on our side". We switched on Wholesale's own sweep on the 14th; before that only the roastery's job moved money automatically, and Wholesale only sent money back by hand now and then. Each account runs its own sweep at :00 and :30 that moves money to whichever sibling account needs it. It is intermittent: most runs are fine, then one call, sometimes two, fails at the same minute. Our job waits five seconds for an answer and then tries again with a new idempotency key, and the second try always works, so nothing was lost so far, but on Wednesday at 10:00 Wholesale paid the roastery $1,250 twice. We think your API applies our retries twice. On the 1st, finance also had two reversals fail with the same error during the month-end clean-up. Please make transfers between our accounts reliable: our jobs should not have to retry them at all, and money must never move twice.

Marta

Their sweep job's own log for Wednesday (Marta pasted it, times UTC):

09:00:00.388  roastery  -> wholesale   250.00 USD  key=sweep-roastery-2026102109      200 in 1.01 s
09:00:00.399  wholesale -> roastery     80.00 USD  key=sweep-wholesale-2026102109     500 in 1.01 s  "Something went wrong on our side."  retry in 5 s
09:00:06.416  wholesale -> roastery     80.00 USD  key=sweep-wholesale-2026102109-retry1  200 in 0.04 s
10:00:00.388  cafes     -> roastery   180.00 USD  key=sweep-cafes-2026102110          500 in 1.01 s  "Something went wrong on our side."  retry in 5 s
10:00:00.509  wholesale -> roastery  1,250.00 USD  key=sweep-wholesale-2026102110     no answer after 5.0 s  retry in 5 s
10:00:06.400  cafes     -> roastery   180.00 USD  key=sweep-cafes-2026102110-retry1   200 in 0.04 s
10:00:10.529  wholesale -> roastery  1,250.00 USD  key=sweep-wholesale-2026102110-retry1  200 in 0.05 s
10:30:00.320  cafes     -> roastery    310.00 USD  key=sweep-cafes-2026102110b        500 in 1.01 s  "Something went wrong on our side."  retry in 5 s
10:30:06.342  cafes     -> roastery   310.00 USD  key=sweep-cafes-2026102110b-retry1  200 in 0.04 s

Support note (Joel, tier 2): I couldn't reproduce it. Transfers in both directions between their accounts work every time when I send them one after the other with the dashboard script. On our side nothing was deployed on the 14th; the only recent infrastructure change is the PostgreSQL 17 upgrade (PAY-2380) on the 9th, maybe related? Ops looked at the database log, saw lock waits and suggested we simply raise the lock timeout (it is 10 s today). The "paid twice" case: there really are two $1,250 transfers from Wholesale to the roastery on Wednesday, about four seconds apart. Engineering pulled the API request log and the database log for Wednesday morning (attached).

Attachments: attachments/paylane-requests-2026-10-21.log (every API request, 08:50–10:35 UTC), attachments/postgres-2026-10-21.log (PostgreSQL server log, same window).

Ask: Transfers and reversals between the accounts of an organization must go through whichever direction they run in and however many of them run at the same moment, without the client having to retry, never moving money twice for one request (idempotency keys keep working as documented), and never overdrawing an account. Support was also asked for a switch that pauses transfers platform-wide while sweeps run: that is not acceptable, other merchants' payments and transfers must not slow down because of Northwind's. Anything that works today must keep working.