What happens if a lot of cache expires at the same time?
Some techniques we can use to reduce or prevent a cache stampede.
In my last note I had a problem. What happens if a lot of cache expires at the same time?
Suppose we have many cache entries with the same TTL. If they all expire at the same time, like 12 AM, a large number of requests can experience cache misses at the same time. Then all requests may query the database, which can put a huge amount of pressure on the database. This problem is called a cache stampede (also known as the thundering herd problem).
There are several techniques we can use to reduce or prevent it.
Jitter
One approach is to add jitter to the cache TTL.
Instead of setting every cache key to have exactly the same TTL, we add a random value to the base TTL. Then expiration times are different, and the cache keys are less likely to expire at exactly the same time.
Singleflight
This is another approach. Suppose 1000 requests are trying to read the same key at exactly the same time, and that key has just expired.
Without singleflight, all the requests are going to hit the database. With singleflight, only one request hits the database, and once that request finishes, the other requests use that data. While the first request is processing, the others are concurrently waiting.
This is useful when many requests are asking for the same cache key simultaneously.
Stale While Revalidate
For very hot keys, we can use stale while revalidate.
In this approach, if the cache expires, instead of immediately treating it as a cache miss, we can temporarily send stale data while refreshing the cache in the background.
But this approach is only appropriate when serving slightly stale data is acceptable.
Proactive Cache Refresh
In this approach, we refresh the cache before it expires. This kind of approach is useful to avoid a cache miss completely.
So, at the end of the day, my idea is that there is no single solution that should always be used. The right approach depends on the system’s requirements.
I’m not going to master all of these things. But understanding why we need them and what trade-offs each one introduces is very important.
That’s it for today, and God bless.