THE SAMVIT LABS JOURNAL
All articles

Quantisation Without Guesswork: What 4-Bit Models Save and What to Measure

A plain-language explanation of quantisation, group metadata and calibration, with reproducible storage arithmetic and a practical quality-testing plan.

On this page

ABOUT THE EVIDENCEThe quantiser and storage format in the figures are deliberately simplified analytical examples. They do not reproduce AWQ, GPTQ or a measured production model format.

An eight-billion-parameter model needs sixteen decimal gigabytes if every weight occupies sixteen bits. Store each weight in four bits and the ideal payload falls to four gigabytes. That is an attractive starting point for running useful models on more modest hardware.

The word ideal matters. Real formats include scales, packing rules and sometimes higher-precision tensors. Running the model also needs memory for attention state, intermediate values and software buffers. Four-bit weights describe one part of the deployment, not the size of the whole process.

Quantisation is best treated as an engineering change with a quality budget. The aim is to reduce resource use while preserving the behaviour the application needs.

Replace many possible values with a smaller set

A trained weight is a number that helps determine the model’s output. Quantisation represents that number using a smaller set of possible values. A scale tells the runtime how to interpret the stored code.

For a toy example, take weights between minus one and one. Set a scale s=1/7s=1/7, round to an integer code and reconstruct the value:

q=clip⁡(round⁡(w/s),−7,7),w^=sqq=\operatorname{clip}\left(\operatorname{round}(w/s),-7,7\right), \qquad \hat{w}=sq

This uses fifteen symmetric levels, leaving one of the sixteen possible four-bit codes unused. It is a teaching example, not the definition of a production four-bit format. A weight of 0.31 reconstructs to approximately 0.2857 because the stored code is two. The reconstruction error is about minus 0.0243.

A staircase approximation rounds weights between minus one and one to fifteen symmetric reconstructed values using a scale of one seventh.
Figure 1. A simplified symmetric quantiser. Nearby weights can share the same stored value. Real formats may use asymmetric ranges, learned scales or non-uniform codes. Download the example values.

Small numerical errors do not translate directly into the same percentage change in answer quality. Some operations tolerate approximation well; others amplify it. That is why file-size arithmetic needs to be followed by task evaluation.

Not every weight matters equally

The AWQ paper studies weight-only quantisation using activation information to identify important channels and choose scaling. Its central lesson is that simply rounding every weight with one global rule leaves useful information on the table. The deployment benefit depends on the matching inference implementation as well as the compressed values. [1]

GPTQ takes a different route: it uses approximate second-order information to reduce reconstruction error during post-training weight quantisation. Both are examples of methods that do more than save an array with fewer bits. Their published results belong to their tested models and hardware, not to every file carrying a four-bit label. [2]

A useful evaluation compares the same base checkpoint before and after compression. Changing the checkpoint, prompt template and quantisation method together makes it hard to identify which change caused a regression.

Account for the metadata

Many formats divide weights into groups. Each group shares metadata such as a scale or zero point. Smaller groups can represent local value ranges more closely, but they also need more metadata.

Consider an invented format containing four payload bits per weight and thirty-two metadata bits per group. If NN weights divide exactly into groups of size gg, the ideal byte count is:

M=N×48+Ng×328M=\frac{N\times4}{8}+\frac{N}{g}\times\frac{32}{8}

For eight billion weights and a group size of 128, the payload is four decimal gigabytes and metadata adds 0.25 gigabytes. The total is 4.25 GB before headers, alignment, unquantised tensors and runtime memory. The effective storage is 4.25 bits per weight for this particular invented format.

For eight billion weights, an invented four-bit format uses 5, 4.5, 4.25 and 4.125 decimal gigabytes at group sizes 32, 64, 128 and 256 respectively.
Figure 2. Analytical metadata overhead for the stated format: one sixteen-bit scale and one sixteen-bit zero point per group. Actual formats make different choices. No runtime allocation is included. Download the calculation data.

Do not use the chart to choose a group size on quality grounds. It contains no quality measurements. Its purpose is to explain why the number on a model label can differ from the final file size.

Weight precision and activation precision are separate

A weight-only method can keep activations at higher precision. A different method may quantise both. SmoothQuant studies an eight-bit weight-and-activation approach by moving quantisation difficulty from activations toward weights through a mathematically equivalent transformation before quantisation. [3]

These distinctions affect kernel support and hardware efficiency. “INT4” is incomplete as a deployment description. Record what is four-bit, what remains in floating point and where dequantisation occurs.

The attention KV cache is another separate choice. Its memory grows with context length and concurrent sequences, so a compressed model can still run out of memory under a long-context workload. Measure peak resident memory for representative requests rather than adding up weight files alone.

Make calibration resemble the intended task

Some methods use calibration examples to choose compression parameters. Those examples should reflect the texts the model will process: document structure, language mix, typical context length and important domain terms.

A set of short general conversations may miss the activation patterns of long financial tables. Conversely, calibrating exclusively on one narrow template can give a misleading impression of broad quality. Keep calibration, tuning and final evaluation examples separate.

Maintain a small set of deliberately difficult cases: negative amounts, similar product identifiers, contradictory instructions and required abstentions. Add cases involving uncommon scripts or long inputs when the application needs them. Record the exact calibration dataset version so that a later build can be reproduced.

Evaluate useful outcomes before throughput

A practical comparison should answer several questions in order:

CheckExample measurementWhat it can reveal
Output structureValid required fields and typesFormatting regressions
Task qualityAccepted extraction or classification resultBusiness-relevant changes
Important slicesPerformance by language and document typeConcentrated failures
AbstentionCorrect review routing on ambiguous inputsUnsafe overconfidence
Serving behaviourTTFT, completion latency and peak memoryDeployment cost and responsiveness
StabilityRepeated runs and load testsRare failures and queue pressure

Use paired examples so each configuration sees the same task. Review disagreements rather than relying only on an overall score. A model that loses one percentage point on a broad benchmark may lose much more on the particular field your application uses.

Finally, measure the actual runtime on the target device. Lower weight precision can reduce memory traffic, but packing, conversion and unsupported kernels can consume some of the gain. Report the serving engine, hardware, batch size, context lengths and model format with the result.

The useful deliverable is a tested configuration: a particular checkpoint, compression recipe, runtime and acceptance record. A small file becomes a practical product component only when that complete configuration works for its intended task.

References

  1. 1
  2. 2
    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh · version 1 · 2022

    Back to text
  3. 3

SAMVIT Labs Research

Research & engineering

The SAMVIT Labs research and engineering journal. We examine model efficiency, practical agent workflows and the evidence needed to make informed deployment decisions.

Figure view