What it costs to look inside a served model

The serving overhead of vllm-lens, and how a fork reduces it

vllm-lens captures activations and steers them inside a vLLM server. These operations slow the server. This walkthrough starts from the cost of upstream vllm-lens, shows each change that the fork makes to reduce it, and then states what the fork still costs against unsteered requests and plain vLLM.

Five setups and four request shapes

In eager mode, Python runs each model layer. With CUDA graphs, the GPU replays a recorded graph, which is much faster for decode.

The table shows what each setup can do.

Upstream has two costs

Figure 1:

How the fork reduces the overhead

The fork makes four changes that affect speed. This table shows each change before and after, and the sections below explain them.

Graphs stay on

Vectors are prepared once

Capture can return one row

Figure 2:

Idle layers do no work

Figure 3:

What the fork costs without steering

Figure 4:

Figure 5:

What steering still costs

Figure 6:

The prefix cache

The copy of an inline vector

Figure 7:

Figure 8:

What capture still costs

What the fork gives up

Which setup to use

Method and limits

All measurements

Request rate of every setup and request shape, in req/s: unsteered / steered.

Figure 9:

Memory and load time of every mode.

Memory of the fork and upstream on Laguna-XS.2.

Where each request’s time goes.