What Happens When Requests Arrive Together?

Average demand can fit while a burst still leaves people waiting. Follow unfinished requests from one interval to the next.

14 sections · 4 questions · 1 coding challenge

1 / 14

The arrow keys move between sections too.

Waiting Work Can Become Another Source of Failure

BOTIE

Google’s account of cascading failures describes how overload can create growing queues, consume resources, and trigger timeouts or retries that make the overload worse. Requests waiting for service are still part of the running system.

Our queue has no timeout or retry yet. That simplification lets us see the accumulation itself. The example uses discrete intervals and invented traffic; it is not a measured latency model of a production server.

Sources for this history

Google SRE Book: Addressing Cascading Failures

The Service Was Quiet, Then Everyone Uploaded a Picture

Botie’s image service looks comfortably fast when we average requests over a minute. Then a class uploads its pictures together. The service must either keep those requests somewhere, reject some, or make the class wait.

A queue is the unfinished work we carry forward. We can understand its behaviour without beginning with a probability distribution. Let’s follow a short sequence of arrivals and give the server a fixed amount of work it can complete each interval.

Arrivals enter at the start of an interval, then service occurs. The initial queue is empty.

Step 1 of 2

Carry the unfinished work forward

Unused service capacity cannot be saved for a future burst.

The quiet first interval does not let the second interval complete more than four requests.
Interval	Arrivals	Can complete	End backlog
1	2	4	0
2	7	4	3
3	1	4	0

1
Arrivals
2
Can complete
4
End backlog
0
2
Arrivals
7
Can complete
4
End backlog
3
3
Arrivals
1
Can complete
4
End backlog
0

The quiet first interval does not let the second interval complete more than four requests.

Step 2 of 2

One interval of the queue

Start with what was already waiting, then admit new work before using this interval’s service capacity.

1
Add
Previous backlog and new arrivals
2
Subtract
Requests the server can complete
3
Keep nonnegative
An empty queue has zero backlog

Repeat using the previous result as state.

Work cannot disappear between intervals

The queue component preserves unfinished work between intervals. Largest value finds the worst boundary backlog rather than only the final state.

Will the queue drain after the burst?

Change arrivals or completion capacity. Each interval starts with the unfinished work left by the preceding one.

BOTIE

Botie needs room for unfinished image requests, including work carried from earlier intervals.

Choose an example

Requests arriving per interval

Requests that can finish per interval

Backlog at the end of each interval

Follow the operations

1. Requests arriving per interval
1. 02
2. 17
3. 21
2. Requests that can finish per interval
1. 04
2. 14
3. 24
3. Queue over time
1. 00
2. 13
3. 20
4. Largest value
```
3
```
5. Result
```
3
```

This counts fixed-size requests in discrete intervals. It does not measure individual latency or implement queue limits and retries.

1. Choose what one time step means

Our interval could be a second or a minute, but arrivals and service capacity must use that same interval. Arrivals are counts of requests, not a rate to add directly to a count. Here, requests arrive at the start and the server works afterward.

Changing that ordering changes results near a boundary. A finer simulation might use individual arrival and completion times. The coarse model is useful because we can inspect every step.

We start with an empty queue. If a service already has unfinished work, that initial state must be supplied.

2. Account for the work that remains

Add the previous backlog to new arrivals. The server completes at most its capacity, so subtract that capacity and clamp a negative remainder to zero. A negative queue would mean saving unused processing slots, which the server cannot do.

backlog[t] = max(0, backlog[t−1] + arrivals[t] − capacity[t])
Initial backlog = 0
Interval 1: max(0, 0 + 2 − 4) = 0
Interval 2: max(0, 0 + 7 − 4) = 3
Interval 3: max(0, 3 + 1 − 4) = 0

The burst creates a queue that drains in the next interval. This is a recurrence: the next answer depends on the previous answer.

3. Store state explicitly in Python

A NumPy array can hold the arrivals, but each backlog depends on the preceding one. A short loop makes that dependence clear. An elementwise subtraction would incorrectly pretend each interval starts empty.

The workshop’s Queue over time component performs this recurrence and exposes the entire backlog sequence. Its inputs are aligned arrivals and capacities, so we can model a temporary worker loss too.

A component can contain a repeated operation once we have explained and inspected that operation.

Follow two intervals by hand in code

The variable waiting carries state. Each appended result records the end of an interval.

Python

import numpy as np
arrivals = np.array([5, 0])
waiting = 0
trace = []
for incoming in arrivals:
    waiting = max(0, waiting + int(incoming) - 2)
    trace.append(waiting)
print(trace)

Output

[3, 1]

The second interval starts with three waiting requests, even though no new request arrives.

4. A stable average does not remove a burst

When long-run arrivals exceed sustainable service, a queue generally cannot stay bounded without dropping work. A lower average gives it a chance to drain, but a large burst can still violate a waiting-time requirement. Equality leaves no spare capacity to recover from variation.

Our sequence is finite. Seeing its queue end at zero does not prove that a future traffic pattern will behave similarly. Keep the workload assumptions beside the result.

Measure the maximum backlog and the time to drain, not only the final queue length.

5. Queue length is not an individual latency measurement

A backlog counts requests waiting or unfinished at a boundary. Latency measures time for a particular request. To calculate individual latency, we would need service order, arrival times within the interval, and request durations.

For a first-in-first-out queue with similar jobs, more work ahead usually implies a longer wait. That intuition is useful, but dividing a coarse queue count by a rate is only an estimate under stated assumptions.

We will report backlog directly. Calling it measured p95 latency would claim evidence this simulator does not produce.

6. Decide what happens at a queue limit

Real queues need resource limits. When full, a service may reject new work with a retry policy, defer nonurgent jobs, or run a cheaper fallback. Unlimited retries can send even more requests into an already overloaded service.

For a camera inspection system, an old frame may become useless before it reaches inference. The appropriate response could be dropping stale frames while recording the coverage gap. A support request may instead need durable storage.

Queue policy depends on the task. Our later failure lesson will make the response explicit rather than pretending every request eventually receives a useful prediction.

7. Compare designs using the same arrival trace

Replay the same burst against alternative capacities. Also include a worker outage and a quiet recovery period. These tests help separate a capacity change from a workload change.

The Python function will reject negative or fractional request counts. If jobs have unequal work sizes, replace counts with a meaningful workload measure or simulate jobs individually.

Botie’s construction asks for the maximum unfinished work. You will need the Queue over time component and an operation that inspects its sequence.