What a Power Cut Taught Me About Rate Limiting

Today wasn’t a good day for us. A power failure affected the entire country.

Most of the time, I look at screens. Today, I decided to spend some time learning from a book. I didn’t read for the whole day. Just a little while. But it made me realize how far I had drifted away from books.

The topic I picked was rate limiting.

Before today, if someone asked me what a rate limiter does, my answer would have been simple:

“It controls how many requests a user can send within a given time.”

That answer wasn’t wrong. But as I kept reading, I realized how many questions were hiding behind it.

Where should the rate limiter run? What happens when we have more than one server? And how do we stop two servers from allowing requests against the same remaining limit?

Let me walk through what I learned.

Why do we need a rate limiter?

Before choosing an algorithm or adding Redis, I think we need to answer a simpler question: what problem are we trying to solve?

Suppose a client keeps sending requests to an API. Every request uses something: CPU time, memory, database connections, or perhaps a third-party service that charges us per call.

If we accept requests without limits, one client could consume resources that other users also need.

A rate limiter helps us control that usage. We can use it to protect services from excessive traffic, distribute capacity more fairly, and manage costs.

It can also help reduce some forms of denial-of-service abuse. A denial-of-service attack tries to make a service unavailable by exhausting its resources. This is different from intercepting a request or stealing an access token. Rate limiting can restrict how quickly someone uses a stolen token, but it does not fix the compromised authentication. It is one layer of protection.

So the need for a rate limiter depends on the service, the resources it uses, and the traffic we expect.

Where should we put it?

My next question was whether we should implement rate limiting on the client or the server.

Client-side controls can help prevent accidental repeated requests and improve the user experience. But we cannot rely on them to enforce a limit. A client can be modified or bypassed.

The server needs to enforce the rules.

We could implement the limiter as middleware inside the application. We could also enforce limits at an API gateway, which gives us a common place to control incoming traffic.

At first, I thought that distributed systems should always move rate limiting into the gateway. But that is too simple. A service might still need its own limits to protect an expensive operation. A gateway and service-level limit can serve different purposes.

There is also the choice between using an existing solution and building our own. That depends on our requirements, engineering time, and the responsibility we are willing to take on.

For now, I want to stay focused on rate limiting. API gateways deserve another note.

Which algorithms can we use?

I looked at four algorithms. They approach the same problem in different ways.

1. Token bucket

Imagine a bucket that holds tokens.

The bucket has a maximum capacity, and tokens are added at a configured rate. Each incoming request needs a token. If a token is available, the request can proceed. Otherwise, the limiter rejects or delays it, depending on the design.

For example, suppose the bucket holds ten tokens and receives two new tokens every second. A full bucket allows a burst of ten requests. After that, new tokens become available at two per second.

The bucket capacity controls the burst allowance. The refill rate controls the sustained rate.

That example helped me understand why both settings matter.

2. Leaky bucket

One way to understand a leaky bucket is as a queue that releases requests at a steady rate.

Requests arrive and wait their turn. If the queue becomes full, additional requests are rejected.

This can smooth out bursts, but waiting has a cost too. We need to decide how much delay is acceptable and how much work the queue can hold.

3. Fixed window counter

This algorithm counts requests within a fixed period.

For example, we might allow 100 requests per minute. When a new window starts, the counter resets.

The problem appears near the boundary. A client could send 100 requests just before one window ends and another 100 just after the next begins.

Both windows respect the limit, but the server receives 200 requests within a very short period.

4. Sliding window log

This approach tracks request timestamps and checks how many fall within the preceding interval.

Instead of asking how many requests arrived during a fixed clock minute, it asks how many arrived during the last 60 seconds.

That avoids the fixed-window boundary problem, but storing individual timestamps requires more memory.

There are other approaches too, including the sliding window counter. I don’t want to call one algorithm the best without first understanding the traffic and the tradeoffs.

Where do the rules and counters live?

The next question was about storage.

There are two different things to manage: the rules and the changing state.

A rule might say, “Allow this user 100 requests per minute.” The state tells us how much of that allowance the user has already consumed.

Rules can come from application configuration or a configuration service. Some designs use background workers to refresh them, but that is an implementation choice.

The changing state needs fast access. For a limiter running in one process, local memory may be enough. When multiple instances need to enforce a shared limit, Redis is one possible shared store.

The data we store depends on the algorithm. It might be a counter, token information, or a collection of request timestamps. It is not always the same counter-and-timestamp structure.

Do we reject every request that exceeds the limit?

My first thought was yes. But it depends on what the system promises to do.

For an API request that cannot be accepted because of a rate limit, the server can return 429 Too Many Requests. It may also include a Retry-After header to tell the client how long to wait before trying again.

Some work can be queued for later processing. But that introduces more questions.

How long can it wait? What happens if the queue fills up? How will the client know whether the work was accepted or rejected?

Putting work into a queue does not remove the capacity problem. It gives us another way to manage it, with its own limits.

We can also provide information about limits and remaining allowance through response headers. The exact headers should follow the API’s documented contract so clients know how to interpret them.

What happens when we add another rate limiter?

This was the part that made me think more carefully.

Suppose user A sends one request through limiter A and the next through limiter B.

If each limiter keeps an independent counter, neither sees the complete picture. The user may receive more allowance than we intended.

A shared store can help both limiters work with the same state.

But sharing the counter is only part of the solution.

Imagine there is one request remaining. Both limiters read that value at almost the same moment. Both decide to allow a request, and then both update the counter.

We have now allowed two requests when only one should have passed.

That is a race condition.

We need the check and the update to happen atomically, as one operation that another request cannot interrupt halfway through.

The book mentioned Lua scripts and sorted sets. I needed to separate their roles in my understanding. Redis Lua scripts can make the decision-and-update sequence atomic. Sorted sets are useful for organizing timestamped requests in a sliding window log. The data structure alone does not prevent the race condition.

A shared Redis deployment also leaves us with questions about failures. What should the limiter do when Redis is unavailable? Allow requests, reject them, or apply a local fallback?

I still need to explore those choices.

What I’m taking away from today

I started with one sentence: a rate limiter controls how many requests a user can send.

Now I can see the decisions behind that sentence. We need to choose what to limit, where to enforce it, how to track usage, and what to do when requests exceed the allowance.

I understand the basics better, but I wouldn’t say I have learned everything about rate limiting. My next step is to build a small example and see what happens when concurrent requests arrive.

I also want to look at the API gateway we use at work and understand how its limits are configured. After that, I plan to read about notification systems.

The power failure made today difficult. Still, I’m glad I spent a little time with a book again.

That’s it for today.

God bless!