What is expensive in a small model
In a model this size, the embedding table weighs as much as the layers themselves: every vocabulary entry costs 160 numbers. So we want the smallest vocabulary that still segments Kabyle properly. I trained four BPE tokenizers on the corpus and measured their fertility — tokens per word — on unseen lines.
| Vocabulary | Tokens per word | Note |
|---|---|---|
| 1 024 | 2.642 | smaller than any generic tokenizer |
| 2 048 | 2.297 | on a par with NLLB-200 |
| 4 096 | 2.049 | |
| 8 192 | 1.863 | the one used by full KabyLLM |
| NLLB-200 (256,204 entries) | 2.269 | |
| Qwen3 (151,669 entries) | 2.746 |
The result is worth reading twice: with 2,048 entries, Kabyle is segmented as finely as by NLLB-200 and its 256,204-entry vocabulary — a hundred and twenty-five times larger — and clearly better than by Qwen3. A vocabulary trained on the target language beats a huge vocabulary trained on a hundred others.
A hundred and twenty-three million characters, a few million parameters
The corpus is KabyLLM's: 123.1 million characters of Kabyle, that is 47.9 million tokens with this tokenizer. The model is a Transformer decoder with 4 layers, a width of 160, a 192-token context and embeddings shared with the output head: 1 593 280 parameters.
The metric is bits per character, which brings the loss down to the character and makes two different tokenizers comparable. All the figures below are measured on the same tail of the corpus, never seen in training. A first 4,200-step run on CPU stopped at 2.09. On a T4 GPU, 20,000 steps in 16 minutes bring the nano down to 1.83 bits per character — and the curve was still falling at the last step: the model has not reached its limit.
A figure not to put side by side: my full KabyLLM, 36.3 million parameters, shows 1.98 bits per character, but on 6,600 chat messages after fine-tuning, not on this corpus of books. Same unit, different validation set: lining them up would suggest that a model twenty-three times smaller does better, which nothing here shows.
03 · QuantisationEight bits per weight, and what it costs
For a model to fit in a web page, it must be shrunk again after training. Each tensor is mapped to 8-bit integers with its own scale. The final file weighs 2.07 MB — half of what float16 would take.
And the price? Measured, not assumed: validation loss goes from 3.262 to 3.269, that is +0.21%, and bits per character from 1.83 to 1.84. On the shorter local run, the cost was only +0.07%. This is the kind of trade-off you cannot dodge when the model has to run on the visitor's machine.
04 · What this model is notA small model stays a small model
It must be said plainly: at this size, the model learns Kabyle spelling, morphology and frequent turns of phrase, not knowledge. It answers no questions, holds no facts and contradicts itself happily. What it demonstrates lies elsewhere: that a low-resource language can have a model runnable anywhere — no server, no account, no connection, a classroom included.
The corpus itself leaves traces: it was built by decoding PDFs, and some debris remains — a (cid:13) occasionally shows up in the generations. The model learns what it is given, debris included; cleaning it is the next easy win.
The port is checked like the others, with the weights served in this page: the browser's BPE segmentation is identical to Python's (8 texts out of 8), the key-value cache returns exactly the same logits as a full recomputation, and the gap with PyTorch stays at 8.6e-6. Count 8 ms per token.