A faster 4-bit layer for our own adapters

The question: can the same model write faster on the same GPU?

Aditya's general model is a 24-billion-parameter model stored in a 4-bit format called NF4, with an adapter trained on top of that 4-bit base. Most of the time a user waits is the model writing its answer, one token at a time. Every token needs a pass over all the model's weights, so the speed of the 4-bit matrix multiply decides how fast answers arrive and how many people one GPU can serve.

Converting the base to a different 4-bit format would allow other fast kernels, but the adapter was trained against NF4 specifically. A different base shifts the numbers the adapter was fitted to, and an earlier serving route that changed the base silently erased most of an adapter's effect. So the constraint was strict: be faster while computing the same NF4 values.

Where the time goes, measured

A profile of the production server with 16 users writing at once settles where the time goes. Fused 4-bit matrix multiplies take 76.6 percent of GPU time. Separate 4-bit unpacking takes 5.4 percent, the adapter's own kernels 4.3 percent and attention only 2.4 percent. Attention, the part most often discussed, is not the bottleneck here; the weight multiplies are.

Our first hypothesis was wrong, and the profile is how we know. We had expected the 4-bit library to unpack every weight into a full-size copy at each step, because that is what its older version does when several users are served together. The production server uses a newer version with a fused kernel, so that waste was not there. An early comparison against the older version had overstated our gain about threefold; every number on this page is measured against what production actually runs.

The hardware floor

With 16 users, each writing step spends roughly 30 milliseconds in the fused 4-bit multiplies. The 4-bit weights are about 12 GB, and this GPU reads memory at roughly 1.8 TB per second, so a single pass over them needs about 7 milliseconds. The production kernel therefore moves weights at around 400 GB per second, well under a quarter of what the memory system allows.

That gap is the opportunity, and it is also a caution. A kernel can be meaningfully faster than production and still far from the limit, so a speed-up is reported as measured, never as a fraction of the theoretical floor.

What the kernel does differently

Each NF4 weight is a 4-bit code that indexes a fixed table of 16 values, scaled once per block of 64 weights. Our kernel reads two codes per byte and decodes both with a single lookup into a 256-entry table that holds the pair of values directly, at full 32-bit precision so the result is identical to the library's own table. The scale is loaded once per block instead of once per weight, and the decoded weights stay in registers inside the multiply rather than being written back to memory.

When a layer has too few output tiles to keep every GPU core busy, the long input dimension is split across more cores and the partial sums are combined at the end. The best tile sizes for each layer shape were measured offline, because the server records its GPU work as fixed graphs at start-up and cannot tune itself while recording.

Exactness on the real model

On real Sarvam-M weights with the real v13d adapter, across five layer types, the kernel's output differs from the 4-bit library's by 0.17 percent, the rounding limit of the 16-bit output format. The adapter changes each layer's output by 1.5 to 4.6 percent, ten to twenty-five times more than that difference, so the adapter's behaviour is preserved. A test that swaps the two halves of each byte must fail, and does, so the tests can see a decoding error.

One layer at a time

Against the production kernel, on a quiet GPU, the new kernel is 1.20 to 1.75 times faster per layer with 16 users and 1.15 to 2.86 times faster with 32. Across one full layer's multiplies the time falls from 0.883 to 0.581 milliseconds, 1.52 times faster. With a single user the production kernel's dedicated single-request path is about twice as fast, so single-user steps stay on it.

Inside the real server

The first attempt to switch the kernel on inside the server measured nothing. The server loads a cached, pre-compiled copy of the model that calls the original operation directly, so our change was never reached: zero calls, and an honest 1.00 times. The working version replaces the operation's implementation itself, and the server's logs confirm the kernel handled every recorded multi-user step.

End to end, generated tokens per second rose from 411 to 473 with 16 users (1.15 times) and from 636 to 910 with 32 (1.43 times). One user is unchanged by design. Smaller than the per-layer gain, as expected: the multiplies are about four fifths of the time, and the rest did not get faster.

Does it change the answers?

Not beyond the server's own noise. The same server, run twice without the kernel, gives different text on 32 of 91 maths answers because requests are batched differently each time, and scores 76 and 74. With the kernel it scores 75, and differs from a run without it on 33 texts. So the kernel moves outputs no more than batching already does. A full 1,300-question check is the condition for switching it on in the chat service.

What comes next

Three steps follow from the measurements, in order of expected value. First, fold the adapter's two small multiplies into the same pass over the weights, so the adapter's 4.3 percent of GPU time stops being a separate trip through memory. Second, let requests that use different adapters share one pass over the base weights, so specialist models do not each cost a full read. Third, use this GPU generation's newer tensor-core instructions, which neither kernel uses today and which address the remaining distance to the memory limit.

Each step is gated the same way as this one: exact against the 4-bit library on real weights, faster against what production runs on a quiet GPU, and then measured end to end in the real server with a second unchanged run as the noise floor.

What this does not establish

It is one GPU type, one model and one adapter. The adapter's own kernels still run separately; folding them into the same pass is the next step. Even the new kernel moves weights at roughly 520 GB per second, well short of the hardware's limit, so most of the headroom remains. The speed-ups are for steady writing with several users; long prompts are still processed by the existing path, and a single user sees no change. The kernel is not enabled in the chat service yet.

Sources and related work

The NF4 format and its 16-value table come from the QLoRA work on 4-bit fine-tuning. The kernel is written in Triton. Production serving uses vLLM with the bitsandbytes 4-bit library. Every figure here comes from saved measurement files: a torch profiler trace of the production server, a per-layer benchmark against the production library on a quiet GPU, and two end-to-end runs of the same server with and without the kernel.