System Design
Feature Flag SDK
"Design a feature flag SDK for Android, like LaunchDarkly or Firebase Remote Config."
Analyzing the Problem
LaunchDarkly and Firebase Remote Config are feature flag SDKs: a remote on/off switch for already-shipped code, decoupling deploy from release so a feature can be turned on without a new build. The SDK evaluates typed flags (getBoolean, getString, getInt, getJson) from any thread, applies targeting rules against developer-supplied user attributes, computes sticky percentage rollouts deterministically, picks up changes within seconds of the console flipping them, honors a global kill switch, buffers exposure events for A/B analysis, and persists the last-known-good flag set across kill, reboot, and reinstall. Only the client SDK is in frame; the targeting console that decides those values is not.
A flag SDK is a value read from anywhere, at any time, that a server can change underneath you. Three terms name the mechanics that makes that safe:
- SSE (Server-Sent Events) - a long-lived, one-way HTTP stream. The client opens an ordinary
GETand the server holds it open, writing events down it as they happen. One direction only, which is all a push needs, and because it is plain HTTP the proxies and CDNs in the path pass it through untouched. Each event carries an id, so a client that reconnects names the last one it saw and gets only the gap replayed. @Volatileand the snapshot swap - reading a field from many threads without a lock is only safe if the writer's store is guaranteed visible to every later read; otherwise a thread can sit on a stale cached copy indefinitely. Marking a reference@Volatilegives that guarantee. Swap a whole immutable object behind it rather than mutating in place, and every reader sees either the entire old flag set or the entire new one - never a half-applied mix, and never a lock on the read path.- Last-known-good cache - flags must answer before the network does, so the last set that worked is kept on disk and restored at launch. The SDK is never blocked on a round trip to be correct; it is only ever more or less fresh.
Why this shape? Because the read is the constraint. getBoolean() is called from a render tree, so it cannot block, cannot throw, and cannot wait for I/O - which forces every expensive thing (network, disk, parsing) off the read path and onto a background one, leaving the reader with nothing but a pointer dereference.
Requirements
Clarifying Questions
| Question | Answer |
|---|---|
| Server-evaluated or client-evaluated targeting? | Client-evaluated - the SDK runs targeting rules against user attributes on device |
| Refresh model - push, poll, or both? | Both - server can push, client polls as fallback |
| Exposure tracking for A/B? | Yes - record which variant each user saw |
| Kill switch in scope? | Yes - a global switch that disables a flag regardless of targeting |
Functional Requirements
- Evaluate typed flags via
getBoolean / getString / getInt / getJsonfrom any thread - Apply targeting rules against developer-supplied user attributes
- Compute sticky percentage rollouts deterministically from user ID + flag key
- Pick up a flag change within seconds when the server can reach the device, and within minutes when it can't
- Honor a global kill switch that disables a flag regardless of targeting
- Buffer and upload exposure events for A/B analysis
- Persist last-known-good flag set across kill, reboot, and reinstall on user data
Non-Functional Requirements
- Evaluation cost
getBoolean()<10 µs amortised; no I/O, no locks on the hot path - Fail-openAny internal exception returns the developer-supplied default; SDK never throws
- Init cost
flags.initialize()<3 ms wall-clock; first eval succeeds before first frame - Push latency<5 s from console to client while the device is reachable
- Fallback cadence15 min default; configurable; respects battery constraints
- Binary sizeCore <300 KB shrunk; no kotlin-reflect
API Design
Public API
Flags.initialize(context, sdkKey = "client-abc123") {
user("u_123") { // immutable User snapshot
attribute("plan", "premium")
attribute("country", "DE")
attribute("install_age_days", 42)
}
bundledDefaults(R.raw.flags_defaults_json) // R.raw = Android build-time resource bundle; see dd1
streaming(true) // SSE on; falls back to polling
pollInterval(15.minutes)
onUpdate { changed -> // optional callback for live UI refresh
if ("checkout_v2" in changed) refreshCheckoutScreen()
}
}
// Evaluation - typed, default-required, fail-open.
val newCheckout = Flags.getBoolean("checkout_v2", default = false)
val theme = Flags.getString("ui_theme", default = "system")
val limit = Flags.getInt("rate_limit_rpm", default = 60)
val config = Flags.getJson<PricingConfig>("pricing_config", default = PricingConfig.SAFE)
Flags.identify(newUser) // attribute change; re-evaluates locally
Flags.refresh() // force a sync; fire-and-forgetThe default argument is also the type witness and the fail-open value - if the SDK can't evaluate (no cache, exception, kill switch), the default is what flows. See the fail-open and bootstrap deep dives below.
High-Level Design
Architecture
The core axis is execution context, one lane per context: init at the top seeds the snapshot, the lock-free eval-time read sits under it, the background refresh lane below that updates it, and the server sits at the bottom. Read and refresh run concurrently around two pieces of shared state - the on-disk cache (the lifetime handoff that lets a cold start evaluate flags before any network call returns) and the in-memory @Volatile snapshot (the cross-thread handoff that lets any thread read without a lock). Both straddle the Eval/Refresh boundary rather than sitting inside either lane, because the contexts that write them are never the context that reads them. Arrows are data flow: flag values move from their source to the eval call site, which passes the user ID on to the bucketer.
The evaluation path reads a single @Volatile FlagSnapshot reference - no locks, no I/O. The refresh path pulls flag sets from the server (SSE push, poller backstop), merges them, and publishes atomically by swapping the reference and persisting to the cache. The cache sits between refresh and eval-time because it is the contract between execution contexts that don't share a process lifetime: refresh rewrites it on every flag change, and a cold-start eval reads it before the network has answered. Bundled defaults seed the snapshot directly at init, never via the cache; the cache restores the last-known-good set on cold start before any network call returns. Each evaluation also records to an ExposureBuffer that is drained by an UploadWorker to /v1/exposures:batch - the same drain pattern as the analytics pipeline, not shown here because it is a side-output, not part of the flag data flow.
Happy-Path - First Launch with Bundled Defaults
Flags.initialize()registers the user snapshot, sets a@Volatilereference to aFlagSnapshotbuilt fromR.raw.flags_defaults_json, and returns within ~2 ms. The app is free to render.- On the SDK's background thread, the cache loader reads the last-known-good flag set from DataStore (Jetpack's on-disk key-value store). If present and not expired, it replaces the snapshot reference atomically (
AtomicReference.set). Flags.getBoolean("checkout_v2", false)reads the current snapshot, finds the flag rule, evaluates against the user attributes, applies sticky bucketing if the rule is a percentage rollout, and returns the variant. Total time: ~5 µs.- The evaluation also calls
exposureBuffer.record("checkout_v2", "variant_A", user.id); the buffer is drained by anUploadWorkerlike the analytics pipeline. - The SSE client connects to
/v1/flags/stream; the server sends a full eval-set (the server-delivered bundle of flag rules) on connect, then a delta whenever a flag changes. - On delta,
FlagSetMergerbuilds a new immutableFlagSnapshotfrom the previous + delta, persists it to DataStore, then publishes it viaAtomicReference.set. - The
onUpdatecallback fires with the set of changed keys so UI code can refresh affected screens. - If SSE disconnects, the periodic
WorkManager(deferred background scheduler) poller runs every 15 minutes and pulls the full eval-set as a backstop - no flag change is lost for more than the poll interval.
Deep Dives
The user opens the app on a plane. No cached flags exist (first launch after reinstall on user data). Every getBoolean() call needs an answer before the first frame renders. Blocking on the network would lock the UI; returning random variants would break determinism.
Three-tier fallback from most-fresh to most-conservative.
Tier 1: server-fresh flag set. If the cache loader found a recent (TTL within 24 h) set in DataStore, the snapshot reference points at it. Used on warm cold starts where the user opened the app before.
Tier 2: stale cache. If the cache exists but is older than the TTL, the SDK still uses it - stale flags beat default-only behaviour for the seconds it takes to reach the server. The exposure event records stale_cache = true so analytics can exclude those from variant comparisons if desired.
Tier 3: bundled defaults. The release engineer ships flags_defaults.json in R.raw at build time. It's a snapshot of every flag and its safe default at the time of build. The SDK loads it from the APK with zero I/O latency (resources are mmap'd by Android). This is what a first-install user gets until the first sync completes.
fun initialize(ctx: Context, sdkKey: String, block: FlagsConfig.() -> Unit) {
val config = FlagsConfig().apply(block)
snapshot.set(FlagSnapshot.from(ctx.resources.openRawResource(config.bundledDefaults)))
scope.launch {
cacheStore.load()?.let { if (!it.isExpired) snapshot.set(it) }
streamer.connect()
}
}The bundled defaults file is regenerated by a CI job after every release branch is cut, so the defaults stay close to current server state. Stale bundled defaults are not a correctness bug (the kill switch and fail-open semantics protect against them) but they shorten the cold-start divergence window.
default argument and the dashboard surfaces the missing key. The developer's default must always be a safe fallback, not the desired new behaviour.getBoolean("checkout_v2", false) is called inside a Compose render tree. An NPE there crashes the whole screen. Any exception path inside the SDK is, transitively, an exception path inside every screen that reads a flag.
The principled approach: if the flag isn't defined, the developer made a mistake; throw FlagNotFoundException to surface it. Reality: a flag added in the server console but not yet in the client's cache (typical during a rollout) crashes every screen that reads it. The "correct" exception is a production outage.
Missing flag → return default. Type mismatch → return default. Now the developer's default argument is essential: it's the fallback when anything goes wrong. The remaining gap is in internal exceptions: a corrupted cache entry, a NumberFormatException inside rule evaluation. Those still propagate.
Wrap the whole get*() body in a try/catch that returns default on any throwable, with a debug-builds-only warning log. Internal logic is allowed to throw freely - the boundary catches it. The exception is captured and dispatched to a dedicated error queue that's uploaded with exposures, so the SDK team sees the failure rate without the host app ever crashing.
fun getBoolean(key: String, default: Boolean): Boolean = try {
val rule = snapshot.get().rule(key) ?: return default // missing key → default
val variant = evaluator.evaluate(rule, currentUser) // may throw on bad rule
exposureBuffer.record(key, variant.id, currentUser.id)
variant.asBoolean() ?: default // type mismatch → default
} catch (t: Throwable) {
errorReporter.record(key, t)
default
}The catch is intentionally broad - Throwable, not Exception. A StackOverflowError inside rule evaluation must not bring down a render frame.
getBoolean() call is one bug away from a global app crash. The discipline is "internal code is allowed to throw; the public boundary catches everything".The user is in the 30% rollout for flag A and the 70% rollout for flag B. Across app restarts, they must see the SAME variant of A (sticky) AND a statistically independent assignment for B (no correlation between flags). Sequential RNG fails both requirements.
Hashing userId alone is the correlated-rollout antipattern. Every flag in a 30% rollout hits the same 30% of users - your experiments are confounded by construction. flagKey in the hash is the one-line fix, and it's the difference between A/B data you can trust and A/B data that tells you nothing.
if (Random.nextDouble() < 0.3) variant_A else variant_B. Each call gets a fresh roll - the user sees A on one screen and B on the next. Useless for product decisions; impossible to A/B test.
bucket = hash(userId) % 100. Sticky across restarts and across flags. But flag A's "in rollout" set and flag B's "in rollout" set are the same users at the same percentages - a user in the bottom 30% sees the new variant of every flag in a 30% rollout. Correlated rollouts pollute experiment data.
The bucket is MurmurHash3(userId || flagKey || salt) % 100. The flag key in the hash decorrelates rollouts: the user's bucket for flag A is independent of their bucket for flag B. The salt rotates per environment so dev/staging/prod don't share buckets - useful when tweaking rollout percentages.
object BucketingHasher {
private const val SEED = 0xC6A4A793.toInt()
fun bucket(userId: String, flagKey: String, salt: String): Int {
val input = "$userId:$flagKey:$salt".toByteArray()
val h = murmur3_32(input, SEED) // MurmurHash3 32-bit
return (h.toLong() and 0xFFFFFFFFL).toInt() % 100 // 0..99
}
}
fun applyRollout(rule: PercentageRollout, userId: String): Variant {
val bucket = BucketingHasher.bucket(userId, rule.key, rule.salt)
return if (bucket < rule.percentage) rule.treatmentVariant else rule.controlVariant
}The treatmentVariant is the new behavior being rolled out; the controlVariant is the unchanged baseline a user outside the rollout falls back to. MurmurHash3 is used because it has well-known uniformity properties at 32-bit output, no cryptographic overhead, and a 5-line implementation. SHA-256 would also work but costs an order of magnitude more per evaluation, and we have no security requirement here - just bucket uniformity.
The ops team flips a flag in the console. Every connected client should respect the change within seconds. Mobile networks drop SSE connections on backgrounding, captive portals, cellular hand-off - so "every connected client" excludes the half that are mid-reconnect.
SSE with exponential reconnect
SSE is the cheapest thing that meets the <5 s kill-switch budget: a WebSocket would buy a back-channel the SDK never uses, and paying for a second protocol to negotiate and keep alive earns nothing here.
OkHttp's EventSource is a thin SSE client. The SDK connects on init, processes event: flag-update messages, and reconnects on error with exponential backoff + jitter. The server uses the Last-Event-ID header (echoed by the SDK on reconnect) to send any updates the client missed during the disconnect window.
val request = Request.Builder()
.url(streamUrl)
.header("Last-Event-ID", lastEventId.toString()) // server resumes from here
.header("X-SDK-Version", sdkVersion)
.build()
EventSources.createFactory(client).newEventSource(request, object : EventSourceListener() {
override fun onEvent(es: EventSource, id: String?, type: String?, data: String) {
lastEventId = id?.toLongOrNull() ?: lastEventId
merger.applyDelta(FlagDelta.parse(data))
}
override fun onFailure(es: EventSource, t: Throwable?, response: Response?) {
scheduler.scheduleReconnect(backoff.next()) // exponential + jitter
}
})Polling as the floor
Even with SSE, a WorkManager job polls /v1/flags/eval-set every 15 minutes. The poll is the floor on update latency when SSE is unreachable (corporate WiFi blocking long-lived connections, captive portal, app backgrounded with battery saver). On Doze/Standby (Android's idle battery-saving modes) the poll cadence drops to the OS-allowed minimum, which is fine - the user isn't seeing the app anyway.
App lifecycle
On ProcessLifecycleOwner (Jetpack's app-wide foreground/background signal) backgrounding, the SDK closes the SSE connection - it's not worth holding a persistent socket while the user isn't in the app, and the OS will tear it down anyway. On foreground, it reconnects immediately with the last event ID. A flag change that fired while backgrounded is delivered as soon as the user returns.
The resume turns on one detail: the client persists only the cursor, never the events. Step through a backgrounding where ops kills a flag while the socket is down, and watch the kill land on reconnect rather than minutes later via the poll.
The cached flag set is the answer the user gets on cold start before any network call returns. A corrupted cache, a partially-written cache, or a cache from a newer SDK version that the current client can't parse must not crash the SDK.
Atomic cache writes
The cache is a single JSON file stored via Jetpack DataStore. DataStore handles atomicity (write-to-temp + rename) and concurrent reads/writes through its serializer interface. A partial write on power loss is rolled back to the previous good state.
Schema versioning per cache record
Every cache entry carries a schemaVersion. On load, if the version is newer than the SDK knows, the SDK discards the cache and reverts to bundled defaults until a fresh sync. Older versions go through a migrator (v1 → v2 → v3). The migrator is in the SDK because the cache is on-device; the server's eval-set always speaks the version the SDK declares in its sync request.
Corruption tolerance
The cache load is wrapped in the same try/catch pattern as evaluation. A JsonParseException returns "no cache" rather than crashing init. The error is buffered like any other internal failure and ships in the next exposure batch.
suspend fun loadOrNull(): FlagSnapshot? = try {
val raw = dataStore.data.first() // suspends; first emission
when (raw.schemaVersion) {
SCHEMA_V3 -> FlagSnapshot.fromV3(raw)
SCHEMA_V2 -> migrateV2(raw)
else -> { errorReporter.record("cache_unknown_version"); null }
}
} catch (t: Throwable) {
errorReporter.record("cache_load_failed", t); null // fall back to bundled defaults
}default argument is the ultimate floor.Tradeoffs
| Decision | Chosen | Alternative | Why |
|---|---|---|---|
| Evaluation location | Client-side from delivered rules | Server-evaluated per request | Latency at eval time is non-negotiable; offline tolerance is free |
| Bootstrap | Bundled defaults + cache + fresh | Default arg only | First launch behaviour matters; bundled defaults bridge the gap |
| Failure model | Fail-open with broad catch | Throw on internal error | Render-tree calls can't throw; default arg is the floor |
| Bucketing hash | MurmurHash3(userId+flagKey+salt) | Hash userId only | Decorrelates rollouts across flags |
| Refresh transport | SSE + 15 min polling floor | Polling only | SSE is the latency win; polling is the correctness floor |
| Kill ordering | Kill checked before targeting and overrides | Kill as a special targeting rule | Ops needs one button that beats everything |
| Exposure dedupe | Per-session, in-memory | No dedupe | Chatty flags would otherwise dominate upload volume |
| Cache schema | Versioned with migrator | Hard-code current schema | SDK versions skew across the install base; cache must tolerate |
Done reading? Mark it so it sticks in your dashboard.