A language model for Kabyle, trained from scratch

A language that fits on a hard drive

Kabyle is a Berber language, and my native language. All usable Kabyle text amounts to about 40 million tokens: three to four orders of magnitude less than what a general-purpose model needs. Part of it sits in some 600 community PDFs, often typeset with pre-Unicode fonts.

Even multilingual models read it poorly. I measured how many tokens their tokenizers need, on average, to write one Kabyle word, over 400,000 characters:

English text sits around 1.3 tokens per word, so Kabyle costs Qwen3 about 2.1 times more, in compute as well as context. Contrary to what I expected, special letters (ɣ, ḍ, ɛ) account for almost none of it (a 1.06 penalty): the cost comes from morphology.

A model that learned by heart

My first version was a classic GPT with 43.27M parameters (8 layers, 128-token context), trained on 2 T4 GPUs on the 995,159 tokens of Kabyle Wikipedia. The tokenizer split on hyphens and lowercased everything: 35,000 distinct words, 42% of which appeared only once, and 6.5% unknown words.

Result: a loss of 1.33 on training and 6.21 on validation. The model was not undertrained, it had memorized its corpus. By the Chinchilla scaling law, 1M tokens justify a model of about 50,000 parameters: mine was nearly 800 times too large.

Decipher before you train

Most of the work went into making the text usable. Three problems specific to Kabyle:

Letters with several forms. Latin ɛ, Greek ε and Cyrillic ԑ stand for the same letter; ɤ often replaces ɣ. Each variant creates duplicate words in the vocabulary. The pipeline normalizes them instead of rejecting them.

Fonts that encrypt the text. Older books are typeset with fonts where $ means ɣ and î means ṭ. It is a substitution cipher, different for each font, sometimes even between the roman and italic faces of the same book. I wrote a decoder that works font by font, and a solver that automatically recovers the table of an unknown font by minimizing the rate of words missing from a Kabyle dictionary.

Words that carry a sentence. Kabyle is agglutinative: yenna-yas ("he told him") holds subject, verb and object. The pipeline keeps these clitics instead of splitting them.

Every bug I hit became a test: 40 regression tests, all taken from real cases.

Pretrain on books, fine-tune on messages

The second version is a Llama-style decoder Transformer, trained from scratch in two stages on a single Kaggle T4.

A real gain, but a modest one

To compare models with different tokenizers, I measure bits per character on the same 6,600 validation messages (lower is better):

Pretraining alone is not enough: the style of messages is too far from that of books. Combined with fine-tuning, it beats the chat-only model by 3.3%. A measurable improvement, not a breakthrough.

Adapt rather than relearn everything

With 40M tokens, a model trained from scratch will never learn to reason. The next route is to adapt a multilingual model: teach it the language and keep what it knows about the world. The pipeline is written and tested: PDF decoding, a filtered and deduplicated corpus, vocabulary extension (targeting about 1.9 tokens per word), continued pretraining, instruction tuning, evaluation and GGUF export to run on a laptop. The budget is estimated at 28 GPU hours, about a week of Kaggle quota; since a T4 supports neither bf16 nor FlashAttention-2, the code detects the card and falls back to fp16 with SDPA attention.

Most of the corpora come from the Kabyle NLP community (Imsidag-community, boffire), Tatoeba, Common Voice and Wikipedia. Giving that community back a merged, normalized corpus would probably be worth more than the model itself.