Data Caching Techniques for High-Volume API Traffic
Serving more than a billion requests a day requires spreading work across many application servers and keeping frequently used data close to them. A key-value cache held in memory or flash can protect databases from repeated reads, but only if it responds quickly, stays available with hot data present, and never serves stale values. Those three requirements pull in different directions, so the client, pool layout and update rules have to be designed together.
Keep cache reads fast by failing quickly
Opening a fresh TCP connection for every cache call adds a handshake and another network round trip. Because an in-memory response may take only about a hundred microseconds while a same-zone round trip is several hundred microseconds, that extra trip can nearly double the observed latency and consume scarce cache network capacity. A client-side pool avoids the handshake: connections are borrowed for one operation and returned afterwards. During busy periods the pool may grow, and surplus connections are closed on return. Tune the initial and maximum pool sizes against peak demand so new connections are not constantly created.
Fail fast instead of retrying
A slow or unresponsive cache node is not a reason to wait or retry. Immediate retries send more work to the node that is already struggling. Delayed retries hold application threads and connections open; at high request rates the queue of waiting work can triple, exhaust memory and then push longer, heavier work into the database. The safer pattern is: use short connection and receive timeouts, mark a failing node down for a few seconds, stop opening new connections to it while it is marked down, and treat the failed read as a miss. Healthy pooled connections may still be used, but no new connection attempt should hide behind TCP timeout behaviour. Failed writes need separate handling, covered below.
Plan pool size and failure impact
Design for node loss from the start. More, smaller nodes reduce the share of the keyspace lost when one disappears, but they make individual hot keys more disruptive. A useful planning rule is to tolerate at least two cache failures without overwhelming the backend. Keep cache and backend sharding independent, so losing one cache node spreads extra reads across backend nodes rather than concentrating them on a single shard. Monitor hit rate and capacity continuously: a restart empties memory, so warming and headroom are operational requirements, not optional extras.
Use fallback pools for outages
A mirrored pool receives every write and remains warm. Give it a different sharding salt from the primary pool; otherwise the surviving node with the same key mapping can receive double load and fail in turn. Mirroring is expensive because hot data is stored twice, but placing one pool in each availability zone keeps most reads local and can offset some cost through lower latency and reduced cross-zone transfer.
Mirrored versus gutter pools
A gutter pool takes the opposite approach. It starts empty and stores only the hottest values for a very short period, such as ten seconds, when a primary node fails. That brief cushion prevents every request from reaching the database while the primary pool recovers. It needs no double writes and can be much smaller, but it protects the backend less than a fully warm mirror. Choose mirroring when backend headroom is tight and availability-zone locality matters; choose a gutter when cost and simplicity matter more.
Reserve capacity with dedicated pools
Memcache-style stores divide memory into slabs by value size to limit fragmentation and evict least recently used entries when space runs out. That gives little control over which datasets survive. Expensive-to-rebuild data, counters and other important values can therefore be pushed out by heavier traffic. Separate pools reserve capacity for selected use cases. Track hit rate per use case before adding pools, because each additional pool also means more open connections from every application server; proxies may become necessary as pool count grows.
Handle hot keys without overloading one node
Configuration values, rate-limit state and data for very large customers can be requested far more often than ordinary keys. Splitting one logical key into several physical keys spreads load, but every client must agree on the split and writes must update all copies, which makes coordination difficult when hot keys change quickly. A simpler option is a small local cache for rarely changing hot values, accepting weaker consistency in exchange for a large reduction in shared-cache traffic. Meta-command support can expose last-access time, making probabilistic detection of hot keys practical without fleet-wide coordination.
Prevent thundering herds
When a popular key expires, is deleted or is evicted, many application servers can miss at once and hit the backend together. Recache and stale policies solve this by letting one client refresh the value while others continue using the existing entry. Marking a key stale on delete triggers the same single-refresher behaviour. If the key is absent and no stale value exists, a lease can still elect one refresher, but the other clients may need to wait, so use that option only when backend overload is the greater risk.
Expand and migrate without cooling the cache
Adding nodes with consistent hashing keeps most keys where they were and moves only the share belonging to new nodes. Removing and later re-adding nodes is different: old values may reappear and create inconsistency, so treat membership changes carefully. For a full replacement, warm the destination gradually. Mirror writes first, then copy a percentage of reads into the destination using add-only population, then read from the destination while writing to both pools, and finally retire the origin. Coordinate each stage by timestamp across servers, keep clocks closely synchronised, and do not replicate non-idempotent counter or lock operations. This staged approach can replace a large cluster in a few hours while keeping hit rates high.
Make consistency a designed workflow
Concurrent application servers create races even when each individual operation looks correct. A reader can load an old database value, another writer can update the cache with a new value, and the reader can then overwrite it with the stale value. Refilling with add instead of set prevents that particular overwrite, but a failed or lost refill can still leave stale data. Meta commands add useful controls: compare-and-swap detects intervening changes, leases elect a single updater, recache policies support stale-while-revalidate behaviour, marking stale avoids delete stampedes, and touch can shorten a TTL without reading and rewriting the value.
Track write failures and refill safely
Do not retry a failed cache write on the request path. Instead, record the key through a write-failure handler, deduplicate reports, and invalidate each affected key until the invalidation succeeds. That makes writes effectively reliable from the perspective of the CRUD logic and simplifies reasoning about races. For reads, try the cache first; on a miss, load from the database and refill with add. A stale refill is usually corrected by the concurrent write that made it stale, and the failure tracker covers the case where that write is lost. If a write succeeds but the key expires immediately and a lagging replica still returns old data, stronger options include embedding keys in database transaction logs for later invalidation, or scheduling a delayed invalidation beyond replica lag.
Update, create and delete across two stores
For an update, shorten the TTL first, then commit the database change, then store the new value with a normal TTL. If the process stops before the database commit, nothing has changed. If it stops after the commit, the shortened TTL makes the stale entry expire quickly. If either cache operation fails, the write-failure tracker invalidates the key once the cache is available. Creates can use a similar lease with a short TTL before the database write, although database-generated unique IDs may make a simple add after the write sufficient. Deletes can shorten the TTL before the database operation and delete afterwards; where refill races are a concern, leave a short-lived deletion marker instead of an empty cache entry.
Decision checklist
Use this checklist before increasing cache traffic: confirm pooled connections and short timeouts are configured; verify failed reads become misses and failed nodes are marked down; size the pool for at least two node failures; choose a mirrored or gutter fallback deliberately; reserve dedicated capacity for expensive datasets; add single-refresher policies for hot keys; use consistent hashing for expansion; rehearse staged migration; and test every CRUD path against lost writes, replica lag and concurrent refills. A cache that is fast, warm and consistent is not a single setting. It is the combined result of client behaviour, pool topology and update discipline.
| Approach | How it works | Best fit | Main trade-off |
|---|---|---|---|
| Mirrored pool | Both pools receive all writes; reads can fall back to the warm mirror | Tight backend headroom or multi-zone locality | Highest memory cost |
| Gutter pool | Small empty pool stores hottest values for a very short TTL | Cost-sensitive outage cushion | Less backend protection than a warm mirror |
Mirrored pools stay warm through double writes; gutter pools hold only short-lived hot values during an outage.
Frequently asked questions
What should a client do when a cache server stops responding?
Treat a failed cache read as a miss and let the application load from its authoritative store. Do not retry immediately or after a short sleep on a high-throughput path: both can multiply in-flight requests, memory use and backend pressure, turning a brief glitch into a cascading slowdown.
How do mirrored and gutter fallback pools differ?
A mirrored pool receives the same writes as the primary pool and stays warm, so reads can move across quickly. A gutter pool is small and empty, holding only very hot values for a short TTL during an outage. Mirroring gives stronger load protection at higher memory cost; a gutter is cheaper and simpler but less effective.
How can database and cache updates stay consistent without a shared transaction?
Use add when refilling after a miss so a concurrent write is not overwritten. For updates, shorten the existing TTL before the database write, then write the new value afterwards. Record failed cache writes and invalidate those keys once the cache is reachable again, so a lost write does not leave stale data behind.
How can a thundering herd be reduced when a hot key expires?
Let one client refresh an expiring or stale key while the others keep using the current value. A lease serves the same purpose when the key is missing entirely, although clients without a stale value may have to wait for the winner to repopulate it.