Every step leaves a trace.

While training and testing our 2B model, we encountered an issue with the RoPE and KV matrices; we addressed this and implemented various enhancements to scale the model up to the 3B level. In our Transformer architecture, we incorporated a "Block Attention Residual" component: across the 36 blocks, for every fourth block, we retained representations generated by previous layer groups as a separate source and combined them with the current token's data using learned weights. We are currently training the model on an 8xH100 cluster on Google Cloud using over 193 billion tokens. We have successfully scaled the model beyond our own consumer-grade GPUs to high-performance GPU clusters. We are pressing ahead tirelessly... You can follow all test results on https://x.com/tanayayitmaz’s X profile.

To build the 2B Base-LM version of our model, we are training it on 57B+ tokens. For this training, we used a different corpus set than the one used for the 0.8B model. It is trained on data from Culturax, Fineweb Eng, Open Subtitles, Math, Legal, Health, QA, and synthetic reasoning. We are training the Base-LM on our own consumer GPUs. It’s a challenging and lengthy process. However, our tests on the current training run are yielding excellent results. You can follow all test results on https://x.com/tanayayitmaz’s X profile.

TanAI has completed its R&D processes, and we are now scaling the model up to 2.0 billion parameters for the productization phase. We are nearing completion of architectural enhancements that will enable a broader context, a larger vocabulary, and improved reasoning and chain-of-thought capabilities.

After the model was scaled up to 0.8B parameters (813.16M), training for the Tokenizer, Encoder, Indexer, Vector, Base-LM, and Condition Projection was completed. During this lengthy process, the model was trained on 16.5B tokens, resulting in a Base model that has learned Turkish very well. In accordance with Chinchilla guidelines, the model was trained with approximately 20 tokens per parameter to ensure it learned the language effectively.
Subsequently, the model was switched to Instruction mode through Condition Stack and SFT training. While the model performs very well in Turkish generation, in Deterministic mode, it successfully generates different responses to the same prompt while remaining consistent with the prompt, thanks to the Cond Proj effect.

All internal tests for the 183M parameter model have been completed. We observed that the model has clearly learned the language and grammar structure. (Turkish) Of course, training with a 106GB corpus and 52K lines of SFT data is very limited, and full-capacity learning is not possible. (The model being very small is a factor.) We are scaling the model up by nearly 4.5X to 813M parameters. We will train it with 700GB of corpus data and 2.56M lines of SFT data. Although these data are large for an 813M Small model, they will clearly improve the model's performance. Based on our experience from previous trainings, we will follow a more stable path. Tokenizer -> Encoder (negative and positive real vector space) -> Offline Indexer -> F16 Vector generation (Fused 256D, ECV 64D, and Chronos 32D) -> Base Model training -> Projection training -> SFT training (with projections) An 813M-parameter model with Adaptive RAG + Context Management may deliver performance comparable to or even better than a 2B+ model.

We decided to enhance the ReasonAUG and ARD (adaptive reason diffusion) features we used in Revision 0 and incorporate them into the Adaptive RAG structure. In ReasonAUG and ARD, data had to be fed statically from an external source. With Adaptive RAG, the model learns more accurate information through a scoring method (such as RLHF) based on the user's data input and the model's responses, and then injects this information into the model. Adaptive RAG can be used in SessionId, Persona, and Global structures. Information learned through a Session passes through the Safety layer and can guide the Model for all users (provided the user accepts this).

The ECV (Emotional Condition Vector) structure was created in an unsorted manner in the L1 layer. When creating the 32-dimensional vector in the ECV projection in the Auto-Regressive core, it will be created in a sorted manner from the 8 different emotions that make up the prompt and the user's general emotional approach.

We solved the problem by changing the training flow. Instead of training both the Base Model and all Projections at the same time, we are training the Base Model first. After the Base Model learns the language structure, we will freeze the Base Model and train the Fused, Evc, and Chronos projections. The Base Model will be trained for 3 days on 2 x 5090 GPUs. While the Base Model is being trained, we are currently feeding the existing projections with 97% 0 vectors and 3% noise so that the Model remembers that the Projections will be used.

During base model training, we noticed that the logits calculations consumed all GPU resources when using Fused(256dim), Evc(32dim), and Chronos(32)dim data consisting of indexed Corpus and .f16 files, and our GPUs started to run out of memory. After overcoming the OOM issue, we were faced with a training time that would take weeks. We are continuing to make improvements to train the model's new Project layers.

1. TanAITokenizer was trained on a 110+GB corpus with a 64K vocabulary in Turkish.
2. TanAIEncoder was trained with TanAITokenizer using 768DIM.
3. The Corpus was tokenized and indexed offline using TanAITokenizer.
4. Using the tokenized structure from TanAITokenizer and TanAIEncoder, 256-vector FUSED, 32-vector ECV, and 32-vector CHRONOS float16 indexes were created.
5. The model's base training is expected to be completed.

TanAIGATCore was coded as a small LLM model. The goal of Revision Zero was to learn Turkish word/sentence structure. The Base Model was created by processing 21.7 billion tokens with a 64K Vocab Customized TanAITokenizer. SFT tests were conducted using the Orca and Aggro datasets to ensure the Model gained a persona.

Model Structure:
  TanAIGATCore(
  (tok_emb): Embedding(64000, 768)
  (dropout): Dropout(p=0.1, inplace=False)
  (blocks): ModuleList(
    (0-11): 12 x GATBlock(
      (norm1): AdaRMSNorm()
      (norm2): AdaRMSNorm()
      (attn): MultiHeadAttentionGQA(
        (q_proj): Linear(in_features=768, out_features=768, bias=False)
        (k_proj): Linear(in_features=768, out_features=256, bias=False)
        (v_proj): Linear(in_features=768, out_features=256, bias=False)
        (o_proj): Linear(in_features=768, out_features=768, bias=False)
        (dropout): Dropout(p=0.1, inplace=False)
      )
      (ffn): SwiGLU(
        (w1): Linear(in_features=768, out_features=3072, bias=False)
        (w2): Linear(in_features=768, out_features=3072, bias=False)
        (w3): Linear(in_features=3072, out_features=768, bias=False)
        (dropout): Dropout(p=0.1, inplace=False)
      )
      (ada_proj): Linear(in_features=768, out_features=3072, bias=False)
      )
    )
  )
  (norm_f): AdaRMSNorm()
  (ada_proj_f): Linear(in_features=768, out_features=1536, bias=False)
  (rotary): RotaryEmbedding()
  (lm_head): Linear(in_features=768, out_features=64000, bias=False)
  (cond_proj): Linear(in_features=256, out_features=768, bias=False)
  (chronos_proj): Linear(in_features=32, out_features=768, bias=False)
  (ecv_proj): Linear(in_features=32, out_features=768, bias=False)
  )
Model Params: 182.72 M

The Prompt Fusion engine architecture has been completed with 18 modules. After the user enters the prompt, its security is enhanced with the PII/Safety module. It is converted to vectors with the Encoder. Low and sufficient context controls have been developed. In low context, ReasonAug modules are activated to bring the prompt to sufficient context capacity. The entire flow is tracked with the State structure and Telemetry.