We present Neurograf v1, an experimental end-to-end Romanian-first language-model pipeline developed to investigate the feasibility of training language models from scratch for a lower-resource European language.
The pipeline comprises RoSfera, a 635 GB Romanian corpus containing 175.7 million documents; an 80k byte-level BPE tokenizer; five decoder-only models ranging from 160M to 6B parameters, trained on 131 billion tokens; three instruction-tuned assistants; and two contrastively trained embedding models.
Training and evaluation were conducted using computing infrastructure at ELI-NP/IFIN-HH.
The model family exhibits consistent scaling of validation loss and increasing Romanian factual and cultural knowledge with model size.
On the 1,000-question WWTBM benchmark, Neurograf-6B achieves 69.89% accuracy on Romanian-context questions, while Neurograf-3.2B reaches 67.38%.
Compute-optimal analysis yields a parameter-scaling exponent of approximately 0.423.
Neurograf v1 serves as an experimental demonstrator rather than a production system.
The study identifies practical challenges involving diacritic preservation, corpus quality, data composition, and tokenization efficiency, while demonstrating the feasibility of developing Romanian-first base models, instruction-following assistants, and semantic embedding models within a unified training pipeline.