Skip to content

Quantisation

Quantisation makes a language model smaller by storing its numbers with less detail. Then a large model can run on an ordinary graphics card, for example for Domino IQ.

Quantisation is a way to make a language model smaller. The model consists of billions of numbers. With quantisation each number is stored with less detail.

An example

A model with 70 billion numbers takes up about 280 GB at full size. Stored with 8 bits it takes up about 70 GB. Stored with 4 bits, which is called Q4, it takes up about 35 GB.

What does it cost?

A smaller model can lose a little quality. How much must be tested in practice. Q4 and Q5 are often seen as good enough for most tasks.

Quantisation and Domino IQ

If Domino IQ is to run the model on your own server, the model must fit in the memory of the graphics card. That is why you almost always use a quantised model. It must be in the GGUF format.

All terms in the glossary