Boris
Boris is OpenCerebral's main family of text models, trained from scratch on public web corpora and published in base, instruction-tuned, and quantized GGUF form. The 1.0 and 1.3 generations are GPT-2 architecture at 75M, 125M, and 250M parameters. The 1.7 generation is a separate architecture experiment, described below.
Architecture
| Boris-1.3-75M | Boris-1.3-125M | |
|---|---|---|
| Architecture | GPT-2 — pre-LN, learned positional embeddings, tied embeddings | |
| Layers / heads / d_model | 12 / 9 / 576 | 12 / 12 / 768 |
| Parameters | 77.4M | 125M |
| Context length | 1024 | 1024 |
| Vocab | 50,304 (GPT-NeoX-20B BPE, padded) | |
| Tokenizer | EleutherAI/gpt-neox-20b | |
| Precision | bf16 autocast with fp32 master weights | |
1.0 and 1.3
The original Boris models (July 2026) were pretrained on FineWeb-Edu alone. Their benchmark results showed a consistent gap: strong enough on the knowledge-flavored tasks, weak on the fluency and coherence tasks — LAMBADA and WinoGrande in particular — which is what a purely educational-text diet produces.
Boris 1.3 is the same base checkpoints extended with continued pretraining passes that mix in DCLM-baseline, each pass with a re-warmed learning rate and each measured individually. The tradeoff discovered along the way is that DCLM improves LAMBADA and WinoGrande but tends to cost ARC-Easy and ARC-Challenge. At 125M, leading with FineWeb-Edu before introducing DCLM avoided that regression. Full pass-by-pass numbers are on the benchmarks page.
Training
| Run | Data | Tokens | Wall-clock (RTX 3060) |
|---|---|---|---|
| Boris-125M base | FineWeb-Edu | 2.50B | 33h 38m 48s |
| Boris-1.3-125M continued | FineWeb-Edu + DCLM-baseline, 7 passes | +2.66B | ~37h |
| Boris-75M base | FineWeb-Edu | 1.55B | — |
| Boris-1.3-75M continued | DCLM-baseline + FineWeb-Edu, 3 passes | +2.40B | — |
Boris-125M's final base-run figures were a loss of 3.2998, a gradient norm of 0.281, and a final learning rate of 6.00e-05. Pass-7 figures for the 1.3 continued run were not preserved and are reported as unavailable on the model card rather than estimated after the fact.
Instruction tuning
The Instruct variants are finetuned on
smol-smoltalk
and OpenAssistant (OASST1) conversational data. Boris-1.3-125M-Instruct took
5.6 hours on one RTX 3060. They ship a chat template, so
tokenizer.apply_chat_template() and llama.cpp's conversation
mode both work without extra configuration. See Usage.
Boris 1.7 — the n-gram table
Boris-1.7-D60M-n30M is a different question from 1.0 and 1.3, not a continuation of them. It puts a 60M-parameter dense trunk next to a 30M-parameter n-gram embedding table, spliced into the residual stream after block 2, and asks whether 30M of lookup buys what 30M of dense weight would at a fixed 2.5B-token budget. The trunk is deliberately conventional so that the table is the only thing under test.
| Boris-1.7-D60M-n30M | |
|---|---|
| Trunk | 10 layers, d=768, 12 heads / 4 KV heads (GQA), SwiGLU 1920, RoPE, RMSNorm, tied embeddings |
| Dense (non-embedding) params | 59,981,568 |
| Embedding params | 25,165,824 (tied) |
| N-gram table | 468,751 rows × rank 64, plus a 64→768 projection = 30,049,216 |
| Context / vocab | 1024 / 32,768 — its own byte-level BPE, trained on this corpus |
| Data | Ultra-FineWeb bulk, then cosmopedia-v2 + FineWeb-Edu anneal over the final 15% |
| Schedule | WSD (2% warmup, 85% stable, 1-sqrt decay); Muon on 2-D linear maps, AdamW elsewhere |
| Run | 2.5B tokens, 10,172 steps at 245,760 tokens/step, 22h 37m, bf16 + torch.compile |
| Final val | loss 3.235, ppl 25.4, 1.0012 bits/char |
The table is keyed on the literal last 2 and 3 token ids — 281,250
bigram rows, 187,500 trigram rows, and one shared out-of-vocabulary row. No
hashing: lookup is an exact searchsorted, one gather per
position and no matmul, so compute per token barely moves. Hit rates on the
finished model run about 65% bigram and 21% trigram.
Ablating the table at inference (--no-ngram) measures how
much the trained model leans on it. That is not the dense control —
that would need its own training run — but it is informative:
| Task | Metric | Intact | Table ablated | Δ |
|---|---|---|---|---|
| HellaSwag | acc_norm | 31.22 | 30.54 | −0.68 |
| ARC-Easy | acc_norm | 45.88 | 43.77 | −2.11 |
| ARC-Challenge | acc_norm | 26.02 | 22.95 | −3.07 |
| PIQA | acc_norm | 62.57 | 60.55 | −2.02 |
| WinoGrande | acc | 51.07 | 50.12 | −0.95 |
| LAMBADA | acc | 26.66 | 25.33 | −1.33 |
| Validation | bits/char | 1.0012 | 1.0699 | +0.0687 |
Removing the table costs 0.069 bits/char and between 0.7 and 3.1 points across the six tasks. The two ARC splits move most; HellaSwag and WinoGrande barely notice. Scored with the model's own tokenizer, on a harness first validated against GPT-Neo-125M's published numbers. It also scores 36.1 ±1.5 on ArithMark-3 (random is 25) and 1022 Elo on BananaMind Base Bench 1.1.
Because 1.7 carries a custom architecture and its own tokenizer, it is not
a transformers checkpoint — load it with the loader in its
repository. Checkpoints are resumable to the byte: optimizer states, data
cursor, and every RNG round-trip through the checkpoint.
Limitations
A model of this size produces text that is frequently inaccurate, inconsistent, or offensive. Boris has received no alignment or safety tuning. Do not use it as a factual reference and do not deploy it without supervision.