Caching Strategies Every System Designer Should Know

Caching is one of the highest-impact techniques in system design. A well-placed cache can turn a 500ms database query into a 5ms in-memory lookup. But caching also introduces complexity: stale data, thundering herds, and cache invalidation.

Why Cache?

| Without Cache | With Cache | |---------------|------------| | Every request hits the database | Hot data served from memory | | O(n) disk reads | O(1) memory reads | | Network round-trips to DB | Local or near-local access | | DB becomes bottleneck | DB load drops dramatically |

Where to Place the Cache

1. Client-Side Cache

• Browser cache, local storage, service workers
• Fastest — no network call
• Limited control over invalidation

2. CDN Cache

• Caches static assets at edge locations
• Great for images, videos, JS/CSS files
• Invalidation strategies: TTL, cache-busting via filenames

3. Application Cache (In-Memory)

• Caches objects inside application memory
• Very fast, but per-instance
• Risk: memory bloat, inconsistent state across instances

4. Distributed Cache

• Shared cache like Redis or Memcached
• Centralized, consistent across instances
• The most common choice for dynamic data

5. Database Cache

• Database internal buffer pool (e.g., PostgreSQL shared buffers)
• Query cache (deprecated in MySQL 8.0)
• You get this "for free" but have limited control

Cache Read Strategies

Cache-Aside (Lazy Loading)

1. App checks cache first
2. If cache miss → fetch from DB
3. Store result in cache
4. Return result

typescriptCopy
async function getUser(id: string) {
  const cached = await redis.get(`user:${id}`)
  if (cached) return JSON.parse(cached)

  const user = await db.users.findById(id)
  if (user) {
    await redis.setex(`user:${id}`, 3600, JSON.stringify(user))
  }
  return user
}

Pros: Simple, cache only contains requested data. Cons: First request always misses; thundering herd risk.

Read-Through

The cache library handles DB lookup automatically. The app only talks to the cache.

Pros: Cleaner app code. Cons: Tight coupling to cache provider.

Refresh-Ahead

The cache predicts which items will be needed and refreshes them before expiration.

Pros: Always fresh hot data. Cons: Complex; can waste resources predicting wrong items.

Cache Write Strategies

Write-Through

1. Write to cache
2. Write to DB synchronously
3. Return success

Pros: Strong consistency. Cons: Slower writes (two operations).

Write-Behind (Write-Back)

1. Write to cache
2. Return success immediately
3. Async job writes to DB later

Pros: Fast writes, high throughput. Cons: Risk of data loss if cache fails before DB write.

Write-Around

1. Write directly to DB
2. Cache is only populated on read

Pros: Avoids polluting cache with infrequently read data. Cons: First read after write always misses.

Eviction Policies

When cache is full, what do you remove?

| Policy | Description | Best For | |--------|-------------|----------| | LRU | Least Recently Used | General purpose | | LFU | Least Frequently Used | Stable popularity patterns | | FIFO | First In First Out | Simple, predictable | | TTL | Time-To-Live expiry | Time-sensitive data | | Random | Random removal | Avoiding specific patterns |

Redis supports LRU, LFU, TTL, and random eviction via maxmemory-policy.

Cache Invalidation Patterns

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton

1. TTL (Time-To-Live)

Set an expiration time. Simple but data can be stale until TTL expires.

bashCopy
EXPIRE user:123 3600

2. Active Invalidation

Delete/update cache entry when DB changes.

typescriptCopy
async function updateUser(id: string, data: Partial<User>) {
  await db.users.update(id, data)
  await redis.del(`user:${id}`)
}

3. Write-Through with Immediate Update

Update cache at the same time as DB. Ensures consistency but couples cache logic with write path.

4. Versioned Cache Keys

Instead of invalidating, change the key version when data changes.

user:123:v1
user:123:v2

Old versions naturally expire via TTL.

Handling Thundering Herd

When a popular cache key expires, thousands of requests hit the DB simultaneously.

Solutions

1. Locking: Only one request rebuilds the cache.
2. Stale-While-Revalidate: Serve stale data briefly while refreshing in background.
3. Jittered TTL: Add randomness to expiration times so they don't all expire at once.

typescriptCopy
const ttl = 3600 + Math.floor(Math.random() * 300) // 3600-3900s
await redis.setex(`user:${id}`, ttl, JSON.stringify(user))

Cache Consistency Trade-offs

| Strategy | Consistency | Performance | Complexity | |----------|-------------|-------------|------------| | Cache-Aside + TTL | Eventual | High | Low | | Write-Through | Strong | Medium | Medium | | Write-Behind | Eventual | Very High | High | | Refresh-Ahead | Strong for hot data | High | High |

When NOT to Cache

• Data changes constantly (cache hit rate too low)
• Data is small and DB is already fast
• Strong consistency is absolutely required
• The cache complexity outweighs the performance gain

Key Takeaways

• Start with cache-aside + TTL for most use cases.
• Use Redis for distributed caching when you have multiple app instances.
• Always plan for cache misses — your system must survive without the cache.
• Add jittered TTL to avoid thundering herds.
• Choose consistency model based on product requirements, not ideology.

A cache is not a database replacement. It's a performance layer. Design your system so that if the cache disappears, everything still works — just slower.