Building a tokenizer for Turkish

On 1 July 2026, Tanay Ayitmaz published “Designing a Tokenizer for a Turkish-Native Model: The TanAI 2B Experience.”

Discussions of language model quality often focus on parameter counts, training data and benchmark scores. This article turns to an earlier step: how the tokenizer represents language before the model begins processing it.

Starting with the language

Turkish is central to the TanAI development process. The article shares the tokenizer perspective from the TanAI 2B experience, placing this foundational component alongside the larger architecture and training story.

The original English article is available on Medium. The site’s model page preserves the development history from the early research revisions through TanAI 3B Ku2.

Read the full article on Medium ↗

Follow the model’s evolution

Read the model history and explore all publications.