Where Does the Prediction Pipeline Run Out of Room?

A fast model can sit inside a slow service. Follow requests through preparation, inference, and storage to find the stage that limits the whole path.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Put Numbers Against the Boxes in a Design

BOTIE

A diagram can connect services without showing whether any of them can handle the proposed workload. Google’s Non-Abstract Large System Design approach develops concrete resource estimates from a system’s requirements, then revises the design as those estimates expose constraints.

Our image service is a small invented example of that process. We will use ideal stage capacities first. Those estimates help choose what to measure, but contention and scheduling overhead mean a real benchmark can disagree.

Sources for this history

Google SRE Workbook: Non-Abstract Large System Design

Why Is the Queue Growing When the Model Is Fast?

Botie’s image service can classify many pictures each second. Yet users are waiting. The model is only one step: each picture must first be decoded, resized, and later written to a result store. Any of those steps can hold up the rest.

We need a small model of the service before buying more hardware. How much work can each stage finish? How quickly does new work arrive? Start with quantities we can compare in the same units.

The model assumes independently pipelined stages and equally shareable work.

Step 1 of 2

Trace one image

The stages can work on different images at the same time, but each image still visits them in order.

1
Prepare
Decode and resize
2
Infer
Run the saved model
3
Store
Write the result

Concurrent stages improve throughput; they do not remove stages from an individual request.

Step 2 of 2

Compare capacities on one scale

Multiply worker count by rate per worker before comparing the stages.

Storage is the limiting stage under these assumptions.
Stage	Workers	Requests per second per worker	Stage capacity
Prepare	2	6	12 requests/s
Infer	1	20	20 requests/s
Store	2	5	10 requests/s

Prepare
Workers
2
Requests per second per worker
6
Stage capacity
12 requests/s
Infer
Workers
1
Requests per second per worker
20
Stage capacity
20 requests/s
Store
Workers
2
Requests per second per worker
5
Stage capacity
10 requests/s

Storage is the limiting stage under these assumptions.

A request has to pass through every stage

The limiting stage changes as the design changes. Both arrays preserve the same stage order.

What capacity does each stage contribute?

Change worker counts and per-worker rates. The smallest stage capacity limits this ideal pipeline.

BOTIE

Botie has worker counts and measured per-worker rates for preparation, inference, and storage.

Choose an example

Worker count by stage

Requests/s per worker by stage

Ideal pipeline capacity, requests per second

Follow the operations

1. Worker count by stage
1. 02
2. 11
3. 22
2. Requests/s per worker by stage
1. 06
2. 120
3. 25
3. Multiply
1. 012
2. 120
3. 210
4. Smallest value
```
10
```
5. Result
```
10
```

This ideal bound omits shared-resource contention, variable request work, and queueing delays.

1. Distinguish elapsed time from completed work

Latency is how long one request takes from arrival to response. Throughput is how many requests finish per unit time. A pipeline can finish many requests per second while each request spends a noticeable time travelling through it.

Imagine several images at different stages. Preparation can work on the next image while inference processes the current one. That overlap explains why adding all stage durations answers a different question from finding the smallest stage capacity.

Keep both quantities in the design. A throughput requirement alone does not protect a user from a long wait.

2. Turn a worker measurement into a stage estimate

Measure a worker using representative images and the intended model. If one worker completes six requests each second, two independent workers would ideally complete twelve. The word independent matters: workers sharing a saturated disk may not provide twice the rate.

Our arrays use one entry per stage. Multiplying aligned worker counts and rates gives a capacity for each stage, all in requests per second.

Workers = [2, 1, 2]
Per-worker rates = [6, 20, 5] requests/s
Stage capacities = [2 × 6, 1 × 20, 2 × 5] = [12, 20, 10] requests/s

This is an estimate to test. Treat measured per-worker rates as specific to a workload and resource allocation.

Name the stage beside its capacity

Keep names in a separate aligned array so a minimum remains interpretable.

Python

import numpy as np
names = np.array(["prepare", "infer", "store"])
capacity = np.array([8., 15., 11.])
i = int(np.argmin(capacity))
print(names[i])
print(float(capacity[i]))

Output

prepare
8.0

argmin returns a position. Indexing uses that position to recover the stage name.

3. Find the narrowest stage

Every request must pass through storage, so a storage stage that completes ten requests per second prevents the whole pipeline from sustaining more than that rate. Increasing inference capacity alone does not change this bound.

The minimum rule assumes each request visits each stage once and stages can overlap. If two stages share one worker, or some requests visit an expensive stage repeatedly, describe that shared work explicitly before taking a minimum.

Ideal pipeline capacity = min(12, 20, 10) = 10 requests/s
Incoming rate = 8 requests/s
Capacity minus demand = 10 − 8 = 2 requests/s

Positive spare capacity is useful, but it does not prove that bursts or slow images will meet a latency target.

4. Check units before changing the architecture

A rate reported per minute must be converted before comparing it with requests per second. A batch rate is not a request rate until batch size is known. A camera producing frames and a model producing crops may not even use the same unit of work.

For example, one frame can contain several candidate objects. The detection stage’s frame rate and the classifier’s crop rate need a conversion based on crops per frame. An average may hide unusually crowded frames.

Write the unit on the wire. The arithmetic is easy to inspect once each rate refers to the same work.

5. Speed up the part that changes the answer

If storage is limiting, first investigate that stage. More storage workers, a different write pattern, or fewer unnecessary writes may help. Once it improves, preparation may become the new limit.

The same reasoning applies to physical inspection. A model can flag pumps faster than a technician can inspect them. Increasing prediction throughput will not create inspection capacity.

Recalculate after each proposed change. The identity of the limiting stage is a property of the whole current design.

6. Leave room for variation and failures

Operating exactly at an estimated capacity leaves no room for bursts, slow requests, or a worker going away. Choose a target using workload measurements and a service requirement. There is no universal percentage that makes every design safe.

Measure both normal and degraded conditions. If a model server loses one worker, check whether the remaining capacity can keep up and how the service handles the accumulated queue.

Botie’s next workshop follows that queue over time. Here, Python reports all stage capacities as well as the minimum so we can see why a design is limited.