FP64, FP32, and INT8 in Transformers

What is worth noticing

FP64, FP32, and INT8 do not merely change how many bytes a number occupies. They change the values that can be represented, the arithmetic a kernel can use, the memory traffic required, and the error a model may tolerate. “Lower precision is faster” is a hypothesis about a specific model and device, not a universal rule.

The hero is a conceptual view: lower-bit storage reduces representational choices. The exact conversion path and speedup depend on the implementation.

Three representations

Format	Storage per scalar	Typical role
FP64	8 bytes	Numerically demanding scientific work and high-precision reference calculations
FP32	4 bytes	General training, evaluation baselines, and operations that need a wider dynamic range
INT8	1 byte	Quantized weights or activations for supported inference kernels

FP64 and FP32 are floating-point formats: their exponent supports a broad dynamic range while their significand controls precision. INT8 stores one of 256 integer codes. To represent real-valued tensors, quantization also needs a scale and usually a zero point:

$$q=\operatorname{clamp}(\operatorname{round}(x/s)+z,-128,127)$$ $$\hat{x}=s(q-z)$$

Here $x$ is the original value, $q$ the stored signed 8-bit code, $s$ the scale, $z$ the zero point, and $\hat x$ the reconstructed approximation.

A reproducible Python example

This standard-library example isolates representation effects. It is not a benchmark and does not claim to simulate a complete Transformer kernel.

import struct

value = 1 / 10
fp64 = struct.unpack("<d", struct.pack("<d", value))[0]
fp32 = struct.unpack("<f", struct.pack("<f", value))[0]

scale = 0.01
zero_point = 0
int8_code = max(-128, min(127, round(value / scale) + zero_point))
restored = (int8_code - zero_point) * scale

print(f"FP64: {fp64:.17f} ({struct.calcsize('<d')} bytes)")
print(f"FP32: {fp32:.17f} ({struct.calcsize('<f')} bytes)")
print(f"INT8 code: {int8_code} ({struct.calcsize('<b')} byte)")
print(f"INT8 restored: {restored:.2f}")

Output:

FP64: 0.10000000000000001 (8 bytes)
FP32: 0.10000000149011612 (4 bytes)
INT8 code: 10 (1 byte)
INT8 restored: 0.10

The FP32 line differs because the value was rounded into fewer representable bits. The INT8 line is interpretable only with its scale and zero point. A different scale can produce a different rounded value or saturation at the signed range boundary.

Where each format fits

FP64 is rarely the default for Transformer inference because it doubles FP32 storage and many accelerators devote more throughput to lower-precision formats. It remains useful when the problem—not the model’s marketing label—requires the additional precision.

FP32 is a useful reference and a common accumulation or fallback format. Modern training often mixes formats: some tensors use fewer bits while selected reductions, accumulators, or model states remain wider for stability.

INT8 is most compelling when supported kernels turn smaller weights and activations into lower memory bandwidth and higher throughput. Weight-only quantization, dynamic activation quantization, and static quantization make different trade-offs. Matrix multiplication may consume INT8 inputs but accumulate products into a wider integer before rescaling.

Limitations

This comparison omits FP16 and BF16, both important in current Transformer training and inference, because the requested scope is FP64, FP32, and INT8. Accuracy loss is model- and calibration-dependent. A safe evaluation compares task metrics, latency, peak memory, and energy on the target hardware, then checks sensitive layers separately.

Connection to the Notes

The Multi-head attention Note explains the shapes behind attention projections and score tensors. Numeric format changes how those tensors are stored and computed; it does not change the underlying query-key-value definition.