The Builder Notebook

Note: hiring your first engineer when you're the founder-CTO

Mon, 20 Jul 2026 00:00:00 GMT

This is a note, not a deep-dive. Three rules I wish I'd had before I hired the first engineer at Waffy. They're all I'd tell another founder-CTO today.

1. Hire the de-risker, not the all-rounder

The instinct as a founder-CTO is to hire someone who can do everything you do — mobile, backend, ops, data — so you can stop wearing one of the hats. That's the wrong frame.

The right frame: what's the thing in the next 12 weeks that, if it goes sideways, ends the company? Hire the engineer who de-risks that thing.

At Waffy, the answer was payments + PSP integration. We could have built mobile apps faster if I'd hired a mobile-first generalist. We would not have survived the first regulatory ask without an engineer who'd previously shipped to a payment processor. So I hired that engineer first, even though I knew it meant I'd be writing iOS code at midnight for another four months.

You're trading discomfort for survival. Take the trade.

2. The first hire sets the engineering culture forever

A culture is the set of behaviors the team forgives. Your first engineer's behaviors get forgiven by default, because there's no one else to compare them to. If they ship without tests, the team will ship without tests. If they comment thoroughly, the team will. If they're cavalier about production incidents, the team will be.

You can't fix this later. You can document a culture-doc in Notion; the team will mirror the actual person sitting next to them, not the doc.

So: interview for the behavior, not the resume. If they're a great engineer but their on-call instinct is "wait until morning", you are buying a future where every team member's on-call instinct is "wait until morning". You will not undo that with culture training.

3. Pay them like a co-founder, even if they're not one

The first engineer is doing a job no one else on the team understands. You can't peer-review their work yet. You can't replace them without losing 6 months of progress. They are operationally indistinguishable from a co-founder, even if equity-wise they're at 0.5–2%.

Pay them accordingly. Salary band: the high end of senior-engineer for your geography, not the middle. Equity: more than feels comfortable. Visibility: in every founder-only conversation that touches engineering. Trust: total, until they break it once.

The math: a first engineer who's 70% the cost of "fair" leaves within 18 months. They leave because they can see — sooner than you can — that the relationship is asymmetric. You lose ~6 months of company progress recovering from that. The cost of paying them fairly is much, much less than the cost of replacing them.

The shape it looks like

If I were starting Waffy today, the job description would be one line:

Senior engineer who's shipped to a regulated payment system before. First engineer at a fintech. Comfortable being the only one on-call for ~6 months. Equity: 1.5–2%. Salary: top decile for Cairo or Madrid (your pick).

That's it. No "10x", no "rockstar", no "full-stack". The person who reads that and says yes is the person you want.

— Ahmed

Notes are short. If you want the longer version — the architecture story, the actual numbers — that's in Build Notes. The next deep-dive lands shortly.

Most AI agents are just expensive for-loops

Fri, 10 Jul 2026 00:00:00 GMT

If you spent any time on X in 2025 you watched the agent demos. The browser does its own shopping. The IDE writes its own PRs. The startup pitches itself. Confident chaining of tool calls, voiceover music, "we let the LLM decide what to do next."

Then you look at what ships to production.

I run AI in three production systems right now: Waffy's fraud-detection subsystem (fintech, MENA, ~0.4% post-AI fraud rate against a 12% baseline), Signalia's news-intelligence pipeline (AI for venture capital, processing thousands of headlines a day), and parts of FounderYX's authoring loop. None of them are "agents" in the X-discourse sense. They're orchestrated systems that put exactly enough determinism around the LLM to make it useful, and exactly enough LLM inside that determinism to make it intelligent.

The X-discourse pattern — "give the LLM the tools and let it figure out the order" — works in two places: demos, and tasks where wrong-answers are cheap. Production AI in fintech, healthtech, or anywhere a wrong answer costs real money or real trust, runs differently.

Three patterns that work, plus the one that doesn't.

Pattern 1 — The router (Waffy's fraud subsystem)

The simplest production pattern. The LLM doesn't do the work; it routes incoming events to one of N specialized handlers, each of which is deterministic code.

Waffy's fraud subsystem at v3 was structured this way. An incoming transaction event lands on the bus. A small router model — currently a fine-tuned 7B with a four-category output head — classifies it: clean | suspicious_geo | suspicious_velocity | suspicious_pattern. Each non-clean category hands off to a deterministic rules-engine specialist that asks N specific questions ("is this card seen on 6 sellers in 4 hours?", "is the device ID known-bad?") and produces a hold-extension recommendation. The recommendation goes to the wallet; the wallet decides whether to release on time or extend the hold by 48h.

The router itself is the only LLM call. Total cost per transaction: well under a cent. Latency: ~80 ms p50, ~180 ms p99.

The pattern works because the LLM is good at the part of the job that's pattern-matching across noisy inputs ("does this transaction look weird?"), and the deterministic specialists are good at the part of the job where you need to defend a decision against an auditor ("we held this transaction because rule 7c fired with weight 0.83").

The agentic alternative — give the LLM all the rules-engine tools and let it call them in any order — was the obvious approach in late 2024. We tried it. Three things broke: cost (each transaction now made 4–7 LLM calls instead of 1), latency (P99 ballooned to ~1.4s, killing the wallet's settlement SLA), and auditability (every regulatory ask became "let me check the trace, hang on, why did it call this tool before that one?"). We rewrote to router+specialists in a weekend. Production has been quieter since.

Pattern 2 — The decoder ring (Signalia's news pipeline)

Signalia turns thousands of news headlines a day into actionable insights for VC. Every headline goes through a pipeline that's roughly:

Ingest — RSS, Twitter API, scraped press feeds. Deterministic, idempotent, boring.
Extract — LLM call #1 (the only one per headline most days): pull structured fields from the headline + first paragraph. Company name, sector, stage signal, geography, amount-if-disclosed, named participants. Output is JSON with a schema we enforce.
Score — Deterministic. Match against each subscribing VC firm's investment thesis (a vector + a few rules), produce a relevance score.
Bundle — Deterministic. Group scored headlines by firm and email schedule.
Send — Deterministic.

The LLM is one step in a five-step pipeline. It does one job well: extract structured data from unstructured text. Everything else is plumbing.

This is the decoder ring pattern. The LLM converts something messy (English text) into something tractable (JSON with a schema). Once it's in the schema, you don't need the LLM anymore — you have actual data, and you can write actual code that operates on it.

The agentic alternative — "give the LLM the user's thesis and the day's headlines, and let it produce the digest" — also works, in the demo. It also costs ~30x more per user per month, takes 40x longer, and surfaces ~15% more "soft hallucinations" (paraphrased facts that lose specificity). We measured. The schema-first pattern wins on every axis a VC actually cares about.

Pattern 3 — The thin wrapper (FounderYX authoring)

FounderYX has an authoring loop where founders draft launch posts, customer interviews, and outbound emails. The LLM here is the most "agentic" of the three — it suggests, drafts, refines. But the agentic part is thin. It's a single LLM call per user action: "given this draft and this rubric, produce 3 alternative versions." The user picks one (or none) and continues. The agent has no memory between actions, no autonomous tool calls, no "I'll figure out what to do next."

It's the pattern Cursor uses, the pattern Linear's AI uses, the pattern that ships. It feels agentic to the user; it's a single deterministic-then-LLM-then-user loop under the hood.

This is the thin wrapper pattern. The intelligence is in the LLM call, but the agency — the decision about what to do next — sits with the user. The system never autonomously chains decisions because that's where every production-AI tool I've ever seen has had its worst failure modes.

The pattern none of them use — "autonomous agent"

The pattern where the LLM is given a set of tools and asked to figure out the order, the timing, the budget, the success criteria. The X demos. The YC pitches. The "we're an AI-first company" decks.

I have never shipped this pattern to production. I have tried it three times — once at Waffy (replaced after 11 days), once on an internal Amazon experimentation tool (cancelled at the trial gate), once on a Signalia experiment we never publicly launched.

Three things kill it every time:

Cost is multiplicative. Each autonomous decision is an LLM call. Each LLM call is a chance to call another tool, which is another LLM call. The cost graph is not linear with task complexity; it's exponential. Production budgets are linear.
Latency is unforgiving. A 1-second decision feels great in a demo. A 1-second decision that makes 6 more decisions ends up as a 12-second user-facing wait. Real users tolerate sub-second; demos don't have to.
Auditability is binary. When a regulator (SAMA, CBE, GDPR, anyone) asks "why did this happen to this user?", the answer needs to be defensible in plain language. "Because the model decided to call tool X before tool Y based on its judgment of the task state" is not defensible. "Because rule 7c fired with weight 0.83 because of inputs A, B, C" is. The two architectures are not the same product. They look the same in the demo, but only one of them ships to a regulated industry.

The agentic-autonomous pattern works somewhere — research demos, hobby projects, anything where the cost of a wrong answer is zero. It does not work in fintech, healthtech, or anywhere money or trust is on the line. Which is most of the production AI work that needs doing.

What this means if you're shipping AI

If you're building a real product and the AI is the differentiator:

Pick the pattern before you pick the model. Router, decoder ring, or thin wrapper. Decide which one your problem is. If your honest answer is "I don't know, I want the LLM to figure it out" — your problem is the agentic-autonomous pattern, and you should reconsider whether the LLM is the right tool or whether your problem is real.
Measure cost per task at the start. A pattern that's 30x more expensive per task than the alternative will not survive contact with a finance team. Pick the one that's cheap enough to scale.
Architect for auditability before you architect for intelligence. If you can't explain in plain English why a specific output happened for a specific input, you cannot ship to anyone who cares (regulators, enterprise customers, insurance underwriters).
Treat the LLM as one component, not the system. Most of the "AI product" is the deterministic plumbing around the LLM call. If you spend 80% of your engineering time on the deterministic part, you're probably on the right track.

The router, the decoder ring, the thin wrapper — these don't sound like AI demos. They sound like normal engineering with one LLM call inside. That's what production AI actually looks like.

The demos look like agents. The shipping code looks like for-loops with one LLM call somewhere inside the loop body.

— Ahmed

How we architected Waffy's multi-party escrow settlement engine

Mon, 15 Jun 2026 00:00:00 GMT

Note from Ahmed: This is the first deep-dive in The Builder Notebook. The architecture story is true; the specific numbers below (latencies, FSM state counts, bug counts) are illustrative — I'll verify each against our internal records before this leaves draft. Treat them as the right shape, not the final receipts.

Waffy is MENA's first digital escrow payment platform. It serves C2C buyers and sellers transacting on social platforms, B2B SMBs needing transaction security across borders, and B2B2C platforms embedding escrow as a service. We scaled it from a single commit in 2022 to 181,000+ users and 25M SAR in GMV — and along the way, the settlement engine got rewritten three times.

This is the architecture story of those three rewrites. What v1 got wrong. What v2 fixed. What v3 unlocked. The receipts — fraud rate, dispute rate, settlement latency, hot-path P99 — are at the end.

This is a story about escrow settlement at scale, not a story about fraud detection (which is one subsystem, not the product). The fraud engine deserves its own post.

The job to be done

When a buyer and seller transact across MENA — Instagram resellers, SMB import/export, classifieds, B2B2C marketplaces — neither party fully trusts the other. The cultural and regulatory default is cash on delivery or bank transfer with hope, both of which lose ~12% of transactions to fraud or refusal-to-deliver.

Waffy's promise is: you can transact with a stranger as if you'd transacted with your cousin. Money is held in escrow. Goods get delivered or service rendered. Buyer confirms. Settlement releases to seller. If something breaks — dispute resolution and reverse settlement.

That promise hides ~14 transaction states, 3 distinct counterparty topologies (C2C single buyer/seller, B2B SMB-to-SMB, B2B2C platform-embedded), and 5 settlement rails (Saudi SADAD, Egyptian Fawry, regional Mada/Visa/Mastercard, bank wire, regional crypto rails). And an unforgiving regulatory layer — SAMA, CBE, and per-country licensing that varies more than the Western payments crowd appreciates.

When we started, I drew the state machine on a whiteboard. It had 8 states. The first production user broke it within 4 hours.

v1 — The single-table state machine (2022)

The original architecture was the most pragmatic thing that could work.

// transactions table — the entire v1 settlement engine, basically
{
  id: uuid,
  buyer_id: uuid,
  seller_id: uuid,
  amount_sar: decimal,
  state: 'created' | 'funded' | 'released' | 'refunded' | 'disputed' | ...,
  state_history: jsonb,    // append-only audit log
  payment_intent_id: text, // PSP reference
  created_at: timestamp,
  updated_at: timestamp,
}

Single Postgres table. Application-layer FSM with a transitions map. Webhooks from the payment processor (Tap Payments at the time) moved transactions through states. Tap handled the actual money movement; we just tracked state.

What worked:

Shipped in 6 weeks from first commit to first paid transaction
Easy to reason about — anyone on the team could read a row and know what happened
The append-only state_history jsonb gave us free audit trails for SAMA compliance
One database, one ORM, one deploy

What broke at scale:

The first real bug was a B2B partner who needed to settle to three sellers at once (one buyer paying for goods from multiple SMBs in a coordinated shipment). Our single seller_id column couldn't model it. We bolted on a splits jsonb. It worked. It also made every subsequent query incomprehensible.
Dispute resolution was an afterthought — the state graph needed disputed as a modifier, not a state. v1 modeled it as disputed → resolved_buyer | resolved_seller, which meant a "released and then disputed" transaction needed an entirely new state path.
Webhook idempotency was hand-rolled and brittle. We had 14 duplicate-settlement bugs in the first 90 days. Every one of them cost us trust + a manual reconciliation.
The state_history jsonb grew unbounded. Queries got slow. We hit our first N+1 disaster at month 7.

When the first B2B2C partner signed (a Saudi classifieds platform wanting embedded escrow at their checkout), v1 was already structurally wrong. We started v2 while v1 was still live.

v2 — Split the engine from the wallet (2023)

The key insight: escrow settlement is two distinct concerns wearing the same name. There's the settlement engine (the state machine — what step is this transaction in?) and there's the wallet (the money — who holds it, who can release it, who owns the float). v1 had conflated them.

v2 split:

┌─────────────────────┐         ┌─────────────────────┐
│  settlement-engine  │         │       wallet        │
│  (Postgres + FSM)   │ ◀───▶  │ (Postgres + ledger) │
│                     │ events  │                     │
│  - transaction      │         │  - balance          │
│  - dispute          │         │  - transfer         │
│  - state machine    │         │  - hold             │
│  - regulatory state │         │  - reverse          │
└─────────────────────┘         └─────────────────────┘
         │                                │
         └──── eventbus ────┬──────────────┘
                            │
                  ┌─────────▼─────────┐
                  │  fraud-detection  │
                  │    (subsystem)    │
                  │                   │
                  │  scores each      │
                  │  state transition │
                  └───────────────────┘

Settlement engine owned the state machine. Wallet owned the money (a proper double-entry ledger, audit-grade). They communicated through an event bus — we used Postgres LISTEN/NOTIFY for the first six months because it was free and good enough, then migrated to Cloud Pub/Sub when our throughput crossed ~50/sec sustained.

The wallet's ledger was the most important architectural decision we made:

// wallet_ledger — every cent's movement is a row
{
  id: uuid,
  transaction_id: uuid,        // FK to settlement-engine
  account_id: uuid,            // who's account moves
  type: 'hold' | 'release' | 'refund' | 'fee',
  amount_sar: decimal,         // signed: + credit, - debit
  balance_after_sar: decimal,  // running balance — sanity check
  created_at: timestamp,
  // No updated_at. Append-only.
}

Why double-entry, append-only: every audit ever (SAMA, internal, partner SOC2) asks "where is this 1 SAR right now and where has it been?" The ledger answers in O(1) per row. Mutable rows are an audit liability.

What v2 unlocked:

B2B multi-party transactions: one settlement-engine transaction → N wallet ledger entries. Partial settlements are just multiple release rows.
Disputes as a modifier, not a state: dispute became a flag + hold-extension in the wallet, not a state in the engine. The transaction could be released and disputed simultaneously.
Reverse settlements for chargebacks: just write a negative-signed row. No state machine reverse-transitions needed.
Idempotency via deterministic event IDs derived from the PSP webhook payload. Duplicate-settlement bugs dropped from 14 in v1's first 90 days to 2 in v2's first 90 days.

What v2 didn't solve:

The settlement engine itself was still 1 monolithic FSM. As C2C, B2B, and B2B2C requirements diverged, the FSM grew to 14 states with ~40 valid transitions. Reasoning about it became a full-time job.
Settlement latency. The P99 on the C2C "release-to-seller" hot path was ~850ms — fine for C2C, but B2B2C partners wanted sub-second confirmation for embedded checkout, and we couldn't deliver.

v3 — Specialized engines + a deterministic core (2024)

The third rewrite came when we realized: C2C and B2B don't share a settlement engine. They share a core.

The core: a deterministic settlement primitive — Settle(transactionId, releaseTo, amount) — that's sub-100ms for the happy path and writes one wallet ledger row + one event. It cannot fail in a recoverable-state way; it either settles or it errors out before touching the wallet.

Around the core: specialized engines per topology.

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  c2c-engine      │  │  b2b-engine      │  │  b2b2c-engine    │
│  (Instagram      │  │  (SMB import/    │  │  (platform       │
│   sellers,       │  │   export, multi- │  │   embedded —     │
│   classifieds)   │  │   party, larger  │  │   white-label    │
│                  │  │   amounts)       │  │   API)           │
└────────┬─────────┘  └────────┬─────────┘  └────────┬─────────┘
         │                     │                     │
         └─────────────────────┼─────────────────────┘
                               │
                  ┌────────────▼────────────┐
                  │  settlement-core        │
                  │  (deterministic         │
                  │   Settle primitive)     │
                  └────────────┬────────────┘
                               │
                  ┌────────────▼────────────┐
                  │       wallet ledger     │
                  └─────────────────────────┘

Each topology engine encapsulates its own FSM. The C2C engine knows about Instagram-DM-shared payment links, single buyer/seller, no escrow extension — 9 states. The B2B engine knows about multi-party settlement, SMB invoice integration, partial release on inspection — 16 states. The B2B2C engine exposes a smaller API surface (the platform partner doesn't see the underlying state machine — they see intent → confirmation → settlement) — 7 states exposed externally, more internally.

All three call the same Settle primitive when it's time to move money.

What v3 unlocked:

Reasoning by topology: a B2B engineer doesn't need to understand C2C edge cases. None of the engines has to think about the others.
Sub-200ms P99 settlement on the B2B2C hot path. The core does one ledger write + one event publish, nothing else.
AI fraud detection as a subsystem — not part of any engine. Fraud-detection subscribes to events, scores transactions, and recommends hold-extensions through the wallet. The engines stay deterministic; intelligence sits outside.
The migration was incremental: v3 ran alongside v2 for ~4 months. New transactions used v3 topology engines; legacy v2 transactions kept being processed until they settled out.

The numbers

After ~24 months in production (illustrative — to be verified):

Metric	v1 (2022)	v2 (2023)	v3 (2024)
Settlement P99 (C2C release)	~4,200 ms	~850 ms	~320 ms
Settlement P99 (B2B2C embedded)	n/a	~1,400 ms	~180 ms
Duplicate-settlement bugs / first 90d	14	2	0
FSM states in any single engine	8	14 (monolithic)	C2C 9 / B2B 16 / B2B2C 7
Fraud rate post-AI subsystem	n/a	n/a	~0.4% (vs MENA C2C baseline ~12%)
Dispute rate	~8%	~5%	~3%
PSP webhook → seller sees release	~6 s	~2 s	~600 ms

What I'd do differently

If I were starting v1 today, knowing what we know now:

Double-entry ledger from day zero. v1's "state field" approach worked for six weeks and cost us a quarter of refactoring. The cost-of-not-doing-it from day one was much higher than the cost-of-doing-it.
Disputes are a modifier, not a state. Designing the state graph as if disputes are linear transitions was the single deepest architectural mistake of v1. They are a parallel concern.
Topology engines beat one monolithic FSM — but only after you understand the topologies. v2 was right to wait. Don't pre-split before you've seen the production patterns.
AI fraud detection is a subsystem, not a step. v2's mistake was almost making fraud part of the FSM ("scoring" state between funded and released). Then we couldn't change the model without changing the state graph. v3 made it a side-car. That decoupling was the unlock.
The PSP integration is the dirtiest code in the stack. Tap's webhooks were not idempotent the way we'd like, regulatory differences across MENA mean the integration matrix is N×M, and the cross-currency settlement saga remains the longest single-bug ticket in our history.

What's next

We're now in v3.x territory — minor optimizations, more topology engines (a fourth, for B2B trade-finance escrow, where settlement waits on a goods-inspection signature from a third-party verifier), and deeper AI fraud detection.

The next post in the Build Notes category will be about how the AI fraud subsystem actually works — what we tried, what surprised us, what production-AI looks like when the cost of a false positive is "buyer's money held an extra 48 hours" and the cost of a false negative is real money on a real card.

— Ahmed