THE BUILDER NOTEBOOK
🛠 Build Notes

How we architected Waffy's multi-party escrow settlement engine

v1 was a single-table state machine. v3 routes 9 transaction types through a deterministic settlement core with an AI fraud subsystem. Here's what broke between them, and what survived.

10 min read By Ahmed Kammorah

Note from Ahmed: This is the first deep-dive in The Builder Notebook. The architecture story is true; the specific numbers below (latencies, FSM state counts, bug counts) are illustrative — I’ll verify each against our internal records before this leaves draft. Treat them as the right shape, not the final receipts.

Waffy is MENA’s first digital escrow payment platform. It serves C2C buyers and sellers transacting on social platforms, B2B SMBs needing transaction security across borders, and B2B2C platforms embedding escrow as a service. We scaled it from a single commit in 2022 to 181,000+ users and 25M SAR in GMV — and along the way, the settlement engine got rewritten three times.

This is the architecture story of those three rewrites. What v1 got wrong. What v2 fixed. What v3 unlocked. The receipts — fraud rate, dispute rate, settlement latency, hot-path P99 — are at the end.

This is a story about escrow settlement at scale, not a story about fraud detection (which is one subsystem, not the product). The fraud engine deserves its own post.

The job to be done

When a buyer and seller transact across MENA — Instagram resellers, SMB import/export, classifieds, B2B2C marketplaces — neither party fully trusts the other. The cultural and regulatory default is cash on delivery or bank transfer with hope, both of which lose ~12% of transactions to fraud or refusal-to-deliver.

Waffy’s promise is: you can transact with a stranger as if you’d transacted with your cousin. Money is held in escrow. Goods get delivered or service rendered. Buyer confirms. Settlement releases to seller. If something breaks — dispute resolution and reverse settlement.

That promise hides ~14 transaction states, 3 distinct counterparty topologies (C2C single buyer/seller, B2B SMB-to-SMB, B2B2C platform-embedded), and 5 settlement rails (Saudi SADAD, Egyptian Fawry, regional Mada/Visa/Mastercard, bank wire, regional crypto rails). And an unforgiving regulatory layer — SAMA, CBE, and per-country licensing that varies more than the Western payments crowd appreciates.

When we started, I drew the state machine on a whiteboard. It had 8 states. The first production user broke it within 4 hours.

v1 — The single-table state machine (2022)

The original architecture was the most pragmatic thing that could work.

// transactions table — the entire v1 settlement engine, basically
{
  id: uuid,
  buyer_id: uuid,
  seller_id: uuid,
  amount_sar: decimal,
  state: 'created' | 'funded' | 'released' | 'refunded' | 'disputed' | ...,
  state_history: jsonb,    // append-only audit log
  payment_intent_id: text, // PSP reference
  created_at: timestamp,
  updated_at: timestamp,
}

Single Postgres table. Application-layer FSM with a transitions map. Webhooks from the payment processor (Tap Payments at the time) moved transactions through states. Tap handled the actual money movement; we just tracked state.

What worked:

  • Shipped in 6 weeks from first commit to first paid transaction
  • Easy to reason about — anyone on the team could read a row and know what happened
  • The append-only state_history jsonb gave us free audit trails for SAMA compliance
  • One database, one ORM, one deploy

What broke at scale:

  • The first real bug was a B2B partner who needed to settle to three sellers at once (one buyer paying for goods from multiple SMBs in a coordinated shipment). Our single seller_id column couldn’t model it. We bolted on a splits jsonb. It worked. It also made every subsequent query incomprehensible.
  • Dispute resolution was an afterthought — the state graph needed disputed as a modifier, not a state. v1 modeled it as disputed → resolved_buyer | resolved_seller, which meant a “released and then disputed” transaction needed an entirely new state path.
  • Webhook idempotency was hand-rolled and brittle. We had 14 duplicate-settlement bugs in the first 90 days. Every one of them cost us trust + a manual reconciliation.
  • The state_history jsonb grew unbounded. Queries got slow. We hit our first N+1 disaster at month 7.

When the first B2B2C partner signed (a Saudi classifieds platform wanting embedded escrow at their checkout), v1 was already structurally wrong. We started v2 while v1 was still live.

v2 — Split the engine from the wallet (2023)

The key insight: escrow settlement is two distinct concerns wearing the same name. There’s the settlement engine (the state machine — what step is this transaction in?) and there’s the wallet (the money — who holds it, who can release it, who owns the float). v1 had conflated them.

v2 split:

┌─────────────────────┐         ┌─────────────────────┐
│  settlement-engine  │         │       wallet        │
│  (Postgres + FSM)   │ ◀───▶  │ (Postgres + ledger) │
│                     │ events  │                     │
│  - transaction      │         │  - balance          │
│  - dispute          │         │  - transfer         │
│  - state machine    │         │  - hold             │
│  - regulatory state │         │  - reverse          │
└─────────────────────┘         └─────────────────────┘
         │                                │
         └──── eventbus ────┬──────────────┘
                            │
                  ┌─────────▼─────────┐
                  │  fraud-detection  │
                  │    (subsystem)    │
                  │                   │
                  │  scores each      │
                  │  state transition │
                  └───────────────────┘

Settlement engine owned the state machine. Wallet owned the money (a proper double-entry ledger, audit-grade). They communicated through an event bus — we used Postgres LISTEN/NOTIFY for the first six months because it was free and good enough, then migrated to Cloud Pub/Sub when our throughput crossed ~50/sec sustained.

The wallet’s ledger was the most important architectural decision we made:

// wallet_ledger — every cent's movement is a row
{
  id: uuid,
  transaction_id: uuid,        // FK to settlement-engine
  account_id: uuid,            // who's account moves
  type: 'hold' | 'release' | 'refund' | 'fee',
  amount_sar: decimal,         // signed: + credit, - debit
  balance_after_sar: decimal,  // running balance — sanity check
  created_at: timestamp,
  // No updated_at. Append-only.
}

Why double-entry, append-only: every audit ever (SAMA, internal, partner SOC2) asks “where is this 1 SAR right now and where has it been?” The ledger answers in O(1) per row. Mutable rows are an audit liability.

What v2 unlocked:

  • B2B multi-party transactions: one settlement-engine transaction → N wallet ledger entries. Partial settlements are just multiple release rows.
  • Disputes as a modifier, not a state: dispute became a flag + hold-extension in the wallet, not a state in the engine. The transaction could be released and disputed simultaneously.
  • Reverse settlements for chargebacks: just write a negative-signed row. No state machine reverse-transitions needed.
  • Idempotency via deterministic event IDs derived from the PSP webhook payload. Duplicate-settlement bugs dropped from 14 in v1’s first 90 days to 2 in v2’s first 90 days.

What v2 didn’t solve:

  • The settlement engine itself was still 1 monolithic FSM. As C2C, B2B, and B2B2C requirements diverged, the FSM grew to 14 states with ~40 valid transitions. Reasoning about it became a full-time job.
  • Settlement latency. The P99 on the C2C “release-to-seller” hot path was ~850ms — fine for C2C, but B2B2C partners wanted sub-second confirmation for embedded checkout, and we couldn’t deliver.

v3 — Specialized engines + a deterministic core (2024)

The third rewrite came when we realized: C2C and B2B don’t share a settlement engine. They share a core.

The core: a deterministic settlement primitive — Settle(transactionId, releaseTo, amount) — that’s sub-100ms for the happy path and writes one wallet ledger row + one event. It cannot fail in a recoverable-state way; it either settles or it errors out before touching the wallet.

Around the core: specialized engines per topology.

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  c2c-engine      │  │  b2b-engine      │  │  b2b2c-engine    │
│  (Instagram      │  │  (SMB import/    │  │  (platform       │
│   sellers,       │  │   export, multi- │  │   embedded —     │
│   classifieds)   │  │   party, larger  │  │   white-label    │
│                  │  │   amounts)       │  │   API)           │
└────────┬─────────┘  └────────┬─────────┘  └────────┬─────────┘
         │                     │                     │
         └─────────────────────┼─────────────────────┘
                               │
                  ┌────────────▼────────────┐
                  │  settlement-core        │
                  │  (deterministic         │
                  │   Settle primitive)     │
                  └────────────┬────────────┘
                               │
                  ┌────────────▼────────────┐
                  │       wallet ledger     │
                  └─────────────────────────┘

Each topology engine encapsulates its own FSM. The C2C engine knows about Instagram-DM-shared payment links, single buyer/seller, no escrow extension — 9 states. The B2B engine knows about multi-party settlement, SMB invoice integration, partial release on inspection — 16 states. The B2B2C engine exposes a smaller API surface (the platform partner doesn’t see the underlying state machine — they see intent → confirmation → settlement) — 7 states exposed externally, more internally.

All three call the same Settle primitive when it’s time to move money.

What v3 unlocked:

  • Reasoning by topology: a B2B engineer doesn’t need to understand C2C edge cases. None of the engines has to think about the others.
  • Sub-200ms P99 settlement on the B2B2C hot path. The core does one ledger write + one event publish, nothing else.
  • AI fraud detection as a subsystem — not part of any engine. Fraud-detection subscribes to events, scores transactions, and recommends hold-extensions through the wallet. The engines stay deterministic; intelligence sits outside.
  • The migration was incremental: v3 ran alongside v2 for ~4 months. New transactions used v3 topology engines; legacy v2 transactions kept being processed until they settled out.

The numbers

After ~24 months in production (illustrative — to be verified):

Metricv1 (2022)v2 (2023)v3 (2024)
Settlement P99 (C2C release)~4,200 ms~850 ms~320 ms
Settlement P99 (B2B2C embedded)n/a~1,400 ms~180 ms
Duplicate-settlement bugs / first 90d1420
FSM states in any single engine814 (monolithic)C2C 9 / B2B 16 / B2B2C 7
Fraud rate post-AI subsystemn/an/a~0.4% (vs MENA C2C baseline ~12%)
Dispute rate~8%~5%~3%
PSP webhook → seller sees release~6 s~2 s~600 ms

What I’d do differently

If I were starting v1 today, knowing what we know now:

  1. Double-entry ledger from day zero. v1’s “state field” approach worked for six weeks and cost us a quarter of refactoring. The cost-of-not-doing-it from day one was much higher than the cost-of-doing-it.
  2. Disputes are a modifier, not a state. Designing the state graph as if disputes are linear transitions was the single deepest architectural mistake of v1. They are a parallel concern.
  3. Topology engines beat one monolithic FSM — but only after you understand the topologies. v2 was right to wait. Don’t pre-split before you’ve seen the production patterns.
  4. AI fraud detection is a subsystem, not a step. v2’s mistake was almost making fraud part of the FSM (“scoring” state between funded and released). Then we couldn’t change the model without changing the state graph. v3 made it a side-car. That decoupling was the unlock.
  5. The PSP integration is the dirtiest code in the stack. Tap’s webhooks were not idempotent the way we’d like, regulatory differences across MENA mean the integration matrix is N×M, and the cross-currency settlement saga remains the longest single-bug ticket in our history.

What’s next

We’re now in v3.x territory — minor optimizations, more topology engines (a fourth, for B2B trade-finance escrow, where settlement waits on a goods-inspection signature from a third-party verifier), and deeper AI fraud detection.

The next post in the Build Notes category will be about how the AI fraud subsystem actually works — what we tried, what surprised us, what production-AI looks like when the cost of a false positive is “buyer’s money held an extra 48 hours” and the cost of a false negative is real money on a real card.

— Ahmed

← Back to all writing