Boris
Boris is OpenCerebral's main family of text models: GPT-2 architecture, trained from scratch on public web corpora, published at 75M, 125M, and 250M parameters in base, instruction-tuned, and quantized GGUF form.
Architecture
| Boris-1.3-75M | Boris-1.3-125M | |
|---|---|---|
| Architecture | GPT-2 — pre-LN, learned positional embeddings, tied embeddings | |
| Layers / heads / d_model | 12 / 9 / 576 | 12 / 12 / 768 |
| Parameters | 77.4M | 125M |
| Context length | 1024 | 1024 |
| Vocab | 50,304 (GPT-NeoX-20B BPE, padded) | |
| Tokenizer | EleutherAI/gpt-neox-20b | |
| Precision | bf16 autocast with fp32 master weights | |
1.0 and 1.3
The original Boris models (July 2026) were pretrained on FineWeb-Edu alone. Their benchmark results showed a consistent gap: strong enough on the knowledge-flavored tasks, weak on the fluency and coherence tasks — LAMBADA and WinoGrande in particular — which is what a purely educational-text diet produces.
Boris 1.3 is the same base checkpoints extended with continued pretraining passes that mix in DCLM-baseline, each pass with a re-warmed learning rate and each measured individually. The tradeoff discovered along the way is that DCLM improves LAMBADA and WinoGrande but tends to cost ARC-Easy and ARC-Challenge. At 125M, leading with FineWeb-Edu before introducing DCLM avoided that regression. Full pass-by-pass numbers are on the benchmarks page.
Training
| Run | Data | Tokens | Wall-clock (RTX 3060) |
|---|---|---|---|
| Boris-125M base | FineWeb-Edu | 2.50B | 33h 38m 48s |
| Boris-1.3-125M continued | FineWeb-Edu + DCLM-baseline, 7 passes | +2.66B | ~37h |
| Boris-75M base | FineWeb-Edu | 1.55B | — |
| Boris-1.3-75M continued | DCLM-baseline + FineWeb-Edu, 3 passes | +2.40B | — |
Boris-125M's final base-run figures were a loss of 3.2998, a gradient norm of 0.281, and a final learning rate of 6.00e-05. Pass-7 figures for the 1.3 continued run were not preserved and are reported as unavailable on the model card rather than estimated after the fact.
Instruction tuning
The Instruct variants are finetuned on
smol-smoltalk
and OpenAssistant (OASST1) conversational data. Boris-1.3-125M-Instruct took
5.6 hours on one RTX 3060. They ship a chat template, so
tokenizer.apply_chat_template() and llama.cpp's conversation
mode both work without extra configuration. See Usage.
Limitations
A model of this size produces text that is frequently inaccurate, inconsistent, or offensive. Boris has received no alignment or safety tuning. Do not use it as a factual reference and do not deploy it without supervision.