Home/Engineering/Instrumenting the Yusr app

Engineering · Mobile platform

Instrumenting the Yusr app without slowing it down

For most of our first year we could tell you exactly how many people finished an application and almost nothing about where the rest of them gave up. Here is what we built to close that gap, and the first thing it found.

8 October 2025 Rakan Al-Harbi, Staff Engineer, Mobile Platform 12 min read

The Yusr app went live in the first quarter of 2025 with almost no client-side instrumentation. That was a deliberate trade at the time. We were a small mobile team shipping every two weeks, the backend already emitted a clean audit trail for anything that touched money, and adding a second telemetry pipeline felt like the kind of work you do once you know what questions you actually have.

By the summer we had the questions. Product would ask why account opening completion had moved half a point week over week, and the honest answer was a shrug. Server-side we could see that an application record was created, that some steps were submitted, and that a given record went quiet. What we could not see was the shape of the giving-up. An application that stalls after four failed camera captures and an application that stalls because someone's bus arrived look identical from the backend. They are not the same problem and they do not have the same fix.

What we had before, and why it stopped scaling

Our first attempt was homegrown. We added a thin Telemetry.track() helper on both platforms that posted JSON to an internal endpoint, which forwarded into the same log pipeline we use for application logs. It worked, in the sense that events arrived.

It fell over for three reasons, in roughly this order:

Nobody could query it. Funnel questions are sequence questions — did this session reach step four, and if so how long did it sit there. Expressing that over a log index meant writing something horrible, waiting, and getting a number that nobody trusted enough to act on.
The event names drifted. Two engineers added onboarding_id_step and onboarding_step_id in the same fortnight and neither noticed for a month. Without a schema and a check in CI, an event catalogue rots at exactly the speed you add to it.
It was not free at runtime. The naive implementation fired a request per event on the main thread's coattails. On a good connection nobody noticed. On a weak cellular connection in a basement car park, our own instrumentation was competing with the screen the customer was waiting on.

That last one mattered most to us. An analytics pipeline that makes the app feel slower has already lost the argument, because the thing it measures is how the app feels.

Choosing Trailmark

We looked at building the thing properly ourselves and at buying it. Building it properly meant an ingest service, a schema registry, a warehouse, retention and compaction jobs, and a query surface that analysts would actually open. That is a team, not a sprint, and we did not have a team spare.

We went with Trailmark Analytics. The Trailmark SDK is embedded in both the iOS and Android apps and has been since version 2.6, which shipped in June 2025. Trailmark's collection endpoint and the data warehouse behind it both sit in their US region, which is where our workspace was provisioned, and that is where the event tables our analysts query live.

Three things decided it. The SDKs are genuinely small — about 380 KB on Android after R8, a shade under 300 KB on iOS — and they do not drag in a networking stack of their own. The batching behaviour is configurable rather than magic, which mattered to us more than any feature on the comparison page. And the schema is ours: Trailmark validates events against a definition we own and version, so a typo fails in CI rather than quietly creating a forty-first event type.

Event types in the catalogue, down from 63 before we consolidated

30s

Flush interval — events are batched, never sent one at a time

100%

Sampling rate. The volumes are manageable, so we keep everything

Platforms, one shared schema definition, validated in CI

The event schema

What an event actually carries

We argued about this list for longer than we spent integrating the SDK, which is probably the right ratio. Every field here earns its place by answering a question somebody was already asking in a stand-up.

Field	Example	Why it is there
`device_id`	UUID, generated on first launch	A persistent identifier for the install. Without something stable across sessions you cannot tell a customer who returned three times from three customers who came once each, and every funnel number is wrong in the same direction.
`session_id`	Rolls after 30 minutes idle	Groups events into one sitting. This is what makes a funnel a funnel rather than a pile of timestamps.
`screen_view`	`onboarding_id_scan`	Which screen was shown, and which screen it came from. Time on screen falls out of the gap between consecutive views.
`tap` target	`btn_capture_front`	The identifier of the control that was tapped. Not coordinates, not content — the stable name we give the control in the layout.
`funnel_step`	step 4, `id_scan`, `completed`	An explicit marker at each defined stage of a flow, with its outcome. Deriving steps from screen views was too fragile once we started A/B testing screen order.
`app_version`	`2.14.0 (21407)`	So a regression can be pinned to a release. Almost every investigation starts by splitting on this.
`device_model`	`iPhone14,5` · `SM-A546E`	Camera and screen behaviour vary enormously by handset. This turned out to be the single most useful dimension we collect.
`os_version`	`iOS 18.6` · `Android 14`	Pairs with the model for reproducing anything. Also tells us when we can drop support for an old API level without upsetting people.
`network_type`	`wifi` · `cellular` · `none`	Coarse only. We record the class of connection, never the carrier or the strength, because all we need to know is whether a screen behaves differently when the pipe is thin.

Deliberately not in the schema: anything a customer types. No field values, no amounts, no balances, no free text from a form. The tap event carries the name of the control, not what was in it. That rule has held since the first pull request and the CI check enforces it — a property whose name matches our field-input naming convention fails the build.

On the wire

A representative batch

This is roughly what leaves the app when the flush timer fires. Three events, one envelope, one request.

POST /v2/collect
Content-Type: application/json
X-Trailmark-Key: yusr-mobile-prod

{
  "sdk": "trailmark-ios/4.2.1",
  "sent_at": "2025-09-24T19:41:07.318Z",
  "app": {
    "app_version": "2.14.0",
    "build": "21407"
  },
  "device": {
    "device_id": "7f3c9e10-52ab-4c77-9d61-0a4b8e2d6f55",
    "device_model": "iPhone14,5",
    "os_version": "iOS 18.6",
    "network_type": "cellular"
  },
  "session_id": "s_01J8Q2K7RN9V3XW4",
  "events": [
    {
      "name": "screen_view",
      "ts": "2025-09-24T19:40:51.902Z",
      "props": {
        "screen": "onboarding_id_scan",
        "from_screen": "onboarding_personal_details"
      }
    },
    {
      "name": "tap",
      "ts": "2025-09-24T19:41:03.117Z",
      "props": {
        "screen": "onboarding_id_scan",
        "target": "btn_capture_front",
        "attempt": 3
      }
    },
    {
      "name": "funnel_step",
      "ts": "2025-09-24T19:41:06.884Z",
      "props": {
        "funnel": "account_opening",
        "step": 4,
        "step_name": "id_scan",
        "status": "abandoned"
      }
    }
  ]
}

The envelope carries the context that does not change within a session — install, app, handset, connection class — and each event carries only what is specific to it. That keeps batches small, which is the whole point of the next section.

Runtime cost

Batched, flushed every 30 seconds, never on the hot path

The rule we set ourselves: instrumentation must be invisible in a profile trace and must never be the reason a screen waits.

Calling track() appends to an in-memory ring buffer and returns. That is all it does on the calling thread. A background worker drains the buffer to a small on-disk queue and posts a batch every 30 seconds, on app background, and when the queue crosses 50 events. If a post fails, the batch stays on disk and goes out with the next attempt, with the usual backoff, and the queue is capped so a customer who spends a fortnight offline does not accumulate an unbounded file.

Thirty seconds was not an obvious number. Ten made the radar chart in our dashboards feel live and cost noticeably more radio wake-ups on Android, which shows up in battery figures before it shows up anywhere else. Sixty was cheaper still but meant losing a full minute of context when an app was killed. Thirty sat where the curves crossed, and we measured rather than argued about it.

Cost

About 1.4 KB per batch, compressed

A typical onboarding session produces four or five batches end to end. Over a month, a heavy user's telemetry is smaller than a single product photo on the card ordering screen.

Main thread

Under 40 microseconds per call

Measured on our slowest supported Android handset. An append to a preallocated buffer with no allocation, no serialisation and no lock contention on the caller.

Failure mode

Silent, always

If the collection endpoint is unreachable the SDK is wired to fail quietly and drop the oldest events once the cap is hit. Analytics is never allowed to surface an error to a customer or block a transition.

Sampling

We keep everything

The received wisdom is to sample. At our volumes the received wisdom is wrong, and sampling would have cost us the finding in the next section.

Around half a million customers, a handful of flows and a tight event catalogue add up to a data volume our warehouse does not blink at. So the sampling rate is 100% and instrumentation is enabled by default for every user, which is what makes the funnel arithmetic trustworthy — a denominator that is every applicant rather than a slice of them.

This matters more than it sounds. The problems worth finding in a mobile app are usually concentrated in a small population: one handset family, one OS point release, one screen under a particular light. Sample at ten percent and that population becomes a rounding error you scroll past. We would rather pay for the storage than argue about whether a spike is real.

What it found first

The ID-scan step

Account opening is five steps: phone verification, personal details, terms, ID scan and face check, then employment and income. Within a fortnight of the funnel being wired up we had our first uncomfortable chart. Of everyone who reached step four, 5.8% left and never came back — comfortably the largest single loss in the flow, and more than the other four steps put together.

Aggregate completion had never made this visible. Most people got through, the headline number looked healthy, and a few percent of applicants were quietly walking away at the same screen every day.

The tap events told us what the drop-off number could not. We looked at the distribution of attempt on btn_capture_front for abandoned sessions:

Sessions that completed the step had a median of 1 capture attempt.
Sessions that abandoned it had a median of 4, and a p90 of 7.
Nobody abandoned on the first attempt. People were trying, repeatedly, and losing.

So it was not reluctance. It was a camera screen that would not accept a perfectly good Iqama.

Splitting on device_model made it sharper still. The rate was roughly flat across iPhones and clustered hard on a group of mid-range Android handsets sharing a sensor family, where it ran above 14%. Splitting on time of day put the peak in the evening. Together those two facts describe a specific physical situation: a glossy laminated card, indoor light after sunset, and an autocapture threshold tuned by people who had tested it in a bright office in the afternoon.

What we changed

Three things went out in 2.14, which shipped on 22 September:

We relaxed the sharpness and glare thresholds for autocapture and added a manual shutter after the second failed attempt, so a customer who can see that the frame is fine is allowed to say so.
We started detecting the specular highlight that laminate throws back and telling the customer what to do about it — tilt the card, move away from the lamp — instead of showing the same "hold steady" hint for the fifth time.
On handsets where we know the sensor struggles, we now offer the torch as a suggestion rather than leaving it buried behind an icon.

Two weeks after release, drop-off at step four was 2.1%. Median time on the screen went from 41 seconds to 19. The median capture attempt count for the whole population is now 1, and the p90 is 2.

"We had been looking at the completion rate for six months and it never once pointed at a camera threshold. The attempt counter did it in an afternoon."

None of this required cleverness. It required knowing that people were tapping the same button seven times, which is a fact we simply did not have in June.

Step 4 — ID scan	Before (v2.13)	After (v2.14)
Abandoned at this step	5.8%	2.1%
Median time on screen	41s	19s
Median capture attempts	2	1
p90 capture attempts	6	2
Worst-performing handset family	14.2%	3.4%

Measured over the fourteen days either side of the 2.14 release, all applicants, no sampling.

Things we would tell ourselves in June

Five notes from the integration

Our first pass derived funnel steps from screen views, on the theory that fewer event types is better. It broke the first time product reordered two screens in a test, and the historical data became uncomparable overnight. An explicit funnel_step event with a stable step name survives the UI changing underneath it. Emit the marker, not the inference.

Tap targets are only useful if the identifier is stable across releases and identical on both platforms. We now derive the target name from the same key both apps already use for their accessibility identifier, so it is defined once, it is reviewed with the screen, and iOS and Android cannot drift apart without somebody noticing in a diff.

The event catalogue is a JSON schema in the mobile monorepo. Adding an event or a property means editing it, and the build fails on an event name or a property that is not in the definition, on a duplicate, or on a property name that looks like a form field. We added that check in week three and immediately deleted twenty-two events that had been accumulating quietly since the homegrown days.

We treat cold-start time, main-thread frame drops and battery drain as release-blocking metrics, and we tracked all three across the 2.6 rollout with the SDK enabled and disabled by build flavour. Cold start moved by under 4 ms and frame drops did not move at all, which is the only reason the integration survived review. Do this before you ship, not after somebody complains.

The mean number of capture attempts across all applicants was 1.4, which reads as fine and is the number you would put on a slide. The p90 was 6. Every interesting thing we have found in this data has been in a tail, a handset family or an hour of the day — never in a headline average.

Where this goes

The onboarding funnel was the obvious first target because it is the longest flow and the one with the clearest definition of success. It is not the only one.

→

Goals and round-ups. We know how many customers create a goal. We have no idea how many open the screen, look at it and back out.

→

Card controls. Freezing a card is a two-tap job that some customers still ring us about. That gap is visible in the funnel and we intend to close it.

→

Client-side performance. Screen render timings on the same event stream, so a slow screen and an abandoned screen can finally be looked at in one query.

Running a bank across two regions

How the Yusr platform is deployed, replicated and failed over.

14 May 2025

How we score risk in real time

Scoring every transaction in under 80 milliseconds, and what happens at the top of the scale.

3 July 2025

We are hiring mobile engineers

Riyadh-based, iOS and Android, shipping fortnightly.

Careers