Will the Model Fit on the Device?

A model file can fit while its running process runs out of memory. Account for parameters, temporary values, and the rest of the runtime.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Small Devices Made the Whole Inference Engine Matter

BOTIE

MCUNet examined neural-network inference on microcontrollers with tight memory and compute limits. Its design combined choices about the network with an inference engine suited to those limits. A useful model had to fit the way the device actually executed it.

Our budget is much simpler. We count parameter storage, an estimated peak activation cost per example, and fixed runtime memory. We are not reproducing MCUNet’s engine or claiming that this linear estimate describes every network.

Sources for this history

Lin and colleagues: MCUNet (2020)

The File Fits. Why Does Inference Run Out of Memory?

Botie copies a model onto an inspection device. The file is smaller than the available memory, but the process fails when it receives a batch of images. Running a model needs room for more than its saved parameters.

Let’s put names on the pieces of memory before adding them. We will separate what stays fixed from what grows with the batch, then work out how many images the device can handle under a stated estimate.

Small invented byte counts keep the arithmetic readable. All quantities use bytes, not a mixture of MB and MiB.

Step 1 of 2

Give each allocation a place

Some memory is shared by the batch; some grows as we process more examples.

Peak live memory matters more than the sum of every tensor ever created.
Allocation	What it holds	Budget assumption
Parameters	Saved learned values	Count × bytes each
Activations	Intermediate outputs	Peak bytes/example × batch
Runtime	Buffers and other fixed work	Fixed byte estimate

Parameters
What it holds
Saved learned values
Budget assumption
Count × bytes each
Activations
What it holds
Intermediate outputs
Budget assumption
Peak bytes/example × batch
Runtime
What it holds
Buffers and other fixed work
Budget assumption
Fixed byte estimate

Peak live memory matters more than the sum of every tensor ever created.

Step 2 of 2

Compare two batch sizes

For this toy device, parameter storage uses 40 bytes, runtime uses 12, and activations use 8 per example. The limit is 100 bytes.

Batch six exactly fills the estimate and leaves no allowance for errors in it.
Batch	Parameters	Activations	Runtime	Total
2	40	16	12	68
6	40	48	12	100

2
Parameters
40
Activations
16
Runtime
12
Total
68
6
Parameters
40
Activations
48
Runtime
12
Total
100

Batch six exactly fills the estimate and leaves no allowance for errors in it.

Budget for a running model

Each multiplication converts a count into bytes. Adding compatible allocations gives a total we can compare with the limit.

Where does the remaining memory go?

Change parameter count, storage precision, batch size, or runtime estimates. A negative remainder means the proposed allocation exceeds the budget.

BOTIE

Botie must leave room for the model, its intermediate values, and the runtime.

Choose an example

Parameter count

Bytes per parameter

Batch size

Activation bytes per example

Fixed runtime bytes

Memory limit in bytes

Estimated memory remaining (bytes)

Follow the operations

1. Memory limit in bytes
```
100
```
2. Parameter count
```
10
```
3. Bytes per parameter
```
4
```
4. Multiply
```
40
```
5. Batch size
```
2
```
6. Activation bytes per example
```
8
```
7. Multiply
```
16
```
8. Add
```
56
```
9. Fixed runtime bytes
```
12
```
10. Add
```
68
```
11. Subtract
```
32
```
12. Result
```
32
```

The model assumes linear activation scaling and fixed runtime overhead. Actual peak memory and quantised-model quality need measurement.

1. Count bytes for the saved parameters

A parameter is a stored value such as a weight or bias. With a fixed-width representation, multiply the number of parameters by bytes per parameter. Eight-bit values use one byte each; 32-bit values use four. Quantised formats may also store scales and other metadata.

A file can be compressed or contain additional information, so its size need not equal the in-memory parameter allocation. Ask what the runtime loads and in which format.

10 parameters × 4 bytes/parameter = 40 bytes

This toy count is deliberately small. The same dimensional calculation applies to a larger model.

2. Make room for intermediate results

A layer creates outputs that later layers consume. Those intermediate values are called activations. Some can be released or reused; others must remain live for a skip connection or another operation. The peak depends on which values coexist.

We will use an estimated peak activation cost per example and assume it scales with batch size. Real kernels may allocate workspaces or choose different algorithms for a larger batch, so validate that assumption by measuring peak memory.

Summing all layer outputs without considering their lifetimes can overestimate memory. Ignoring them can underestimate it.

3. Keep runtime memory in the budget

The process also needs input and output buffers, library workspaces, and runtime state. For this exercise they are a fixed allocation. That is an accounting assumption, not a claim that runtime overhead is always fixed.

Training adds other allocations, often including gradients and optimiser state. A serving estimate that omits them cannot be reused as a training-memory guarantee.

Estimated bytes = parameter_count × bytes_per_parameter
                  + batch_size × activation_bytes_per_example
                  + runtime_bytes
For batch 2: 10 × 4 + 2 × 8 + 12 = 68 bytes
Headroom = 100 − 68 = 32 bytes

The workshop returns headroom. A negative answer is useful feedback that the proposed design does not fit this budget.

Separate fixed memory from batch memory

A vector of candidate batches lets us inspect several totals without changing the fixed allocation.

Python

import numpy as np
batches = np.array([1, 2, 3])
fixed_bytes = 24
activation_bytes = 6
print((fixed_bytes + batches * activation_bytes).tolist())

Output

[30, 36, 42]

Each entry is a candidate total. These are separate proposed runs, not allocations to add together.

4. Solve for the maximum batch under this model

Subtract fixed allocations from the limit. Divide the remainder by activation bytes per example, then round down because a batch contains a whole number of examples. If the fixed allocations already exceed the limit, even batch zero fails the budget.

Our function returns zero when no positive batch fits. That is a design failure to investigate, not permission to serve an empty batch as a successful request.

Available for activations = 100 − 40 − 12 = 48 bytes
Maximum batch = floor(48 / 8) = 6

Rounding up would exceed the limit. Exact equality fits the arithmetic but leaves no reserve for an inaccurate estimate.

5. Changing precision changes more than storage

Fewer bits can reduce parameter storage and sometimes computation. The model’s numerical behaviour may also change, and some hardware needs particular formats to gain speed. Measure prediction quality and runtime after conversion.

Shrinking parameters may have little effect when activations dominate peak memory. Inspect the allocations before deciding which optimisation is worth trying.

A smaller file is evidence about storage. It is not by itself evidence about latency or task accuracy.

6. Measure the intended workload

Use the real input sizes, batch pattern, runtime version, and device. Variable-length text or larger images can change activation requirements. A warm process may retain allocations, and peak allocation metrics may differ from total device memory use.

Record what the measurement includes and repeat it under representative conditions. Keep a reserve chosen from that evidence rather than relying on the exact boundary of an optimistic estimate.

Botie lets you change one allocation at a time. The Python challenge calculates the maximum batch, then tests that it obeys the limit under the stated linear model.