Home/Engineering/Running a bank across two regions

Platform engineering

Running a bank across two regions

Every request a Yusr customer makes is served out of Riyadh. There is a second region in Frankfurt that mostly sits there being boring. Four times a year we make it do some work, and in April it did more than usual.

22 April 2026 Yazeed Al-Harbi · Staff Engineer, Platform 12 minute read

When you build a bank from scratch you get to make one genuinely nice decision early: where the data lives. We made ours in the second week of 2024, before there was a single line of production code, and we have not had to revisit it since. Yusr's primary region is me-central-1 — our cloud provider's Riyadh region, inside the Kingdom. Every write, every card authorisation, every balance lookup, every push notification originates there. If you have used the app, you have talked to Riyadh.

That is the easy half. The hard half is the question every regulator, every auditor and every one of us at three in the morning eventually asks: and what happens if that region has a bad day?

Our answer is a second region, eu-central-1 in Frankfurt, which holds a continuously updated copy of the estate and is expected to be able to run the read path on its own. This post is about how that is wired, what we promise ourselves in terms of recovery objectives, and what happened when we rehearsed it in April.

me-central-1

Primary region, Riyadh — carries 100% of production traffic

1.4 s

Median replication lag to eu-central-1 over the last 90 days

15 min

Recovery point objective for the core ledger

4 × year

Full game-day failover exercises against live traffic

The shape of it

One region that works, one that waits

We deliberately did not build active-active. A bank ledger that accepts writes in two places at once is a distributed-systems research project, and we would rather ship a card.

Primary

me-central-1 · Riyadh

Three availability zones, all in-Kingdom. Runs the full stack: the core ledger, the customer profile store, the card authorisation gateway, the fraud engine, the document store behind account opening, the queues, the caches, everything the app talks to.

  • All customer writes land here.
  • Synchronous quorum replication across the three zones.
  • mada card authorisation never leaves this region — it is a domestic rail and we keep it that way.
  • Zone loss is survivable with no human involvement and no visible impact.
Secondary

eu-central-1 · Frankfurt

A warm standby. It holds an asynchronous replica of the primary datastore, a mirrored copy of the object store, and a scaled-down but complete set of the services needed to serve reads. It accepts no customer writes and is not in any write quorum.

  • Compute runs at roughly 20% of primary capacity and scales out on promotion.
  • Promotion is manual and requires two on-call approvals. We have never wanted a replica that can promote itself.
  • Backups and long-retention snapshots land here as well as in Riyadh.
  • Exercised on a schedule, so that the first time it serves a customer is never during an incident.

Why Frankfurt and not somewhere closer

The honest answer is availability of the managed services we depend on. When we made the choice in early 2024, eu-central-1 was the nearest region to Riyadh where our provider offered the complete set — the managed Postgres flavour we run, cross-region snapshot copy, the queue service and the key management integration — at the maturity we needed. The round-trip is around 78 ms from Riyadh, which is comfortable for asynchronous replication and merely tolerable for serving reads. We re-open this decision every year.

How the copy stays fresh

The core ledger runs on managed Postgres. Inside me-central-1 we replicate synchronously across zones: a write is not acknowledged to the application until a second zone has it. Across regions we replicate asynchronously, streaming write-ahead log records to Frankfurt as fast as the link allows and never blocking a customer's payment on a packet crossing a continent.

That choice is the whole trade-off. Asynchronous means the Frankfurt copy is always slightly behind, and in a hard failure of the primary we would lose whatever had not yet shipped. So we measure the lag obsessively. Over the last ninety days the median has been 1.4 seconds and the 99th percentile 8.6 seconds, with a single excursion to 71 seconds during a provider network event in February that resolved itself before anyone reached a laptop. Our alarm fires at 45 seconds sustained for two minutes, which pages the platform on-call.

Roughly, the cluster definition looks like this. Real one has more tags and a lot more opinions about parameter groups.

-- platform/datastore/yusr-core.yaml
cluster: yusr-core
engine:  postgres 15.6

primary:
  region: me-central-1              # Riyadh
  zones:  [me-central-1a, me-central-1b, me-central-1c]
  replication:
    in_region: synchronous          # quorum: 2 of 3 zones must ack
  encryption:
    at_rest: kms/yusr-core-primary  # key material stays in me-central-1
  backup:
    continuous_wal:   true
    snapshot_every:   15m
    retention_days:   35

standby:
  region: eu-central-1              # Frankfurt
  replication:
    cross_region: asynchronous      # never in the write quorum
    stream:       wal
  promotion:
    mode:      manual
    approvals: 2                    # platform on-call + incident commander
  encryption:
    at_rest: kms/yusr-core-dr
  alarm:
    replica_lag_seconds: 45
    for: 2m
    page: platform-oncall

objectives:
  rpo: 15m
  rto: 45m

The fifteen-minute RPO is not derived from the replication lag — that would be optimistic to the point of dishonesty. It comes from the snapshot interval, which is the floor we can actually defend if streaming replication itself is what broke. If everything is healthy we expect to lose seconds. Fifteen minutes is the number we are willing to write down.

What we promise, per component

Not everything deserves the same effort. A missing statement PDF for four hours is annoying; a ledger that cannot say what your balance is for four hours is a different category of problem. So the targets are set per component and reviewed by the risk committee each half.

ComponentRPORTOHow it is achieved
Core ledger
Balances, postings, transaction history		 15 min 45 min Async stream to eu-central-1 plus 15-minute snapshots. Manual promotion.
Card authorisation
mada and international scheme rails		 0 < 5 min Active across three zones in me-central-1 only. Domestic rail, in-region by design; no cross-region path.
Customer profile & support records
Contact details, preferences, ticket history		 15 min 45 min Shares the core cluster and the same replication path.
Document store
Onboarding document and image objects		 15 min 4 hours Cross-region object replication to eu-central-1, versioned, separately keyed.
Notification pipeline
Push, SMS and email delivery		 5 min 30 min Queue replicated; delivery provider is region-independent and re-points on failover.
Fraud & risk scoring 1 hour 1 hour Model artefacts mirrored; feature store rebuilds from the ledger on promotion.
Analytics warehouse 24 hours Best effort Rebuilt from source. Nobody is paged for this one.

RTO is measured from the incident commander's decision to fail over, not from the start of the incident. Detection and the argument about whether to pull the handle are separately budgeted at 15 minutes.

Game day

A plan you have never run is a wish

Four times a year we stop treating the second region as an insurance policy and make it earn its keep. January, April, July, October. Same window, same runbook, a different failure injected each time.

The rules are fixed: it happens on a real Saturday morning against real traffic, the status page is updated before anything moves, and nobody on the exercise team is allowed to have looked at the injected failure in advance. A rehearsal against a staging environment with synthetic load tells you your YAML parses. It does not tell you that a service you forgot about has a region baked into a config file.

What happened on 11 April

The April exercise was the fourth we have run and the first with a scope we had been building up to for a year: cut the customer-facing read path over to eu-central-1 and leave it there, serving live traffic, until we had seen enough. Writes stayed in Riyadh throughout — this was never a full promotion, and the primary was healthy the whole time. Balance screens, transaction history, statement downloads, card controls and the in-app help content were all answered from Frankfurt.

We went into it expecting to abort. We did not.

05:58 — Status page updated

T-9 min

A scheduled entry goes up on our status page before a single weight changes. The exercise is logged there as a planned resilience test with a window, and it stays visible until we close it out.

06:07 — Read traffic shifted

T+0

The edge routing weight for read endpoints goes from me-central-1: 100 to eu-central-1: 100 in one step. We debated ramping it and decided a ramp would only teach us about a state we would never actually be in during a regional failure.

06:09 — First customer read served from Frankfurt

T+2 min

Somebody in the Kingdom opened the app and their balance screen was rendered out of eu-central-1. Everyone in the room watched the same dashboard tile flip. It was, in the end, extremely undramatic.

06:11 — Latency settles

T+4 min

p95 read latency moved from 61 ms to 148 ms and stayed flat. Slower, clearly, but well inside the budget where the app still feels immediate. Cache hit rate in the secondary recovered to normal within about six minutes as the warm set filled.

06:22 — The one real failure

Defect found

Statement PDF generation started returning errors. The renderer had a hard-coded object store endpoint pinned to me-central-1 in a config file written eleven months earlier by someone who had since changed teams. Roughly 40 statement downloads failed over four minutes before we pushed a config change. This is precisely the class of bug that only a real exercise finds, and it is the reason we do this on a Saturday instead of writing a document about it.

06:48 — Decision to fail back

T+41 min

We had the data we came for. The runbook allows up to two hours; the incident commander called it early on the grounds that another hour would only produce more of the same graph.

06:50 — Back on primary

Complete

Weights returned to me-central-1. Latency was back to baseline inside ninety seconds. Status entry resolved at 07:05 after we had checked the error budgets.

The headline number: 41 minutes of live customer read traffic served from eu-central-1. In that window Frankfurt answered about 3.1 million read requests across roughly 26,000 customer sessions, at a 99.94% success rate once the statement renderer was fixed. Writes continued to be accepted in Riyadh without interruption for the entire exercise.

We are pleased with that. It is the first time the secondary region has done real work for real people rather than passing a synthetic health check, and the thing we were most afraid of — some quiet coupling to the primary that nobody had documented — turned out to be one stale line in a PDF service.

The command itself is unglamorous, which is the point:

$ yusrctl traffic shift --scope read --to eu-central-1 --reason gameday-2026-04

  plan: read-path-failover
  from: me-central-1 (weight 100 -> 0)
  to:   eu-central-1 (weight 0 -> 100)

  preflight
    [ok]   replica lag 1.2s (threshold 45s)
    [ok]   standby capacity 3.1x current read rps
    [ok]   status page entry INC-2026-041 published
    [ok]   write path unaffected (me-central-1 primary healthy)
    [warn] cache warm set 12% in eu-central-1

  approvals: y.alharbi (platform-oncall), r.almansour (incident-commander)

  applied at 06:07:14 +03  ·  propagation complete 06:08:02 +03

Follow-up

Five things we changed afterwards

Region pinning is now a build failure

A lint rule rejects any literal region string outside the two approved configuration modules. It found four more offenders on the first run, none of which had ever been exercised.

Cache pre-warm before the shift

Twelve percent warm was survivable at six in the morning. It would not be at six in the evening. The runbook now warms the secondary's cache from the replica for thirty minutes before any weight moves.

Latency budget written down properly

We now state explicitly that read-path operation from eu-central-1 is expected to run at roughly 2.4× primary latency, and which screens degrade first if it goes beyond that.

Fail-back rehearsed as its own step

Going back was smooth, but it was smooth because the primary was never actually broken. The July exercise will fail back from a genuinely cold cache.

Clearer status page language

Our first draft of the 11 April entry said "scheduled maintenance", which is technically true and completely uninformative. Customers who look at a status page during a slow app want to know whether something is wrong. It now says what the exercise is, that it is planned, and what they might notice — which, on the day, was nothing at all for the overwhelming majority.

What comes next

The July exercise, on the 18th, raises the stakes: a full promotion of the standby cluster in a sandboxed copy of the estate, timed end to end against the 45-minute RTO, followed by a cold fail-back. If that goes the way April went, the October exercise becomes the one we have been circling since 2024 — promoting the real cluster, accepting writes in eu-central-1, and finding out whether our reconciliation is as good as we think it is.

Longer term we would like a second in-Kingdom zone group with enough separation to serve as the primary DR target, which would let Frankfurt step back to being purely a backup destination. That depends on what our provider builds in me-central-1 and when. We are watching it closely.

Until then: one region does the work, one region waits, and four times a year we make sure the one that waits still remembers how.

Every exercise is on the status page

Planned failover tests, their windows and their outcomes are published alongside incidents. If the app felt slightly slower one Saturday morning, that is where to look.

View system status

More from engineering

How we score risk in real time

Every card authorisation gets a score in under 40 milliseconds. Here is what goes into it and what happens at the top of the range.

3 March 2026

Instrumenting the Yusr app without drowning in events

What we measure in the mobile app, how the pipeline is put together, and the things we decided not to collect.

28 January 2026

We write about the parts of the platform we would want to read about ourselves.

All posts