Boris
Boris is OpenCerebral's main family of text models, trained from scratch on public web corpora and published in base, instruction-tuned, and quantized GGUF form. The 1.0 and 1.3 generations are GPT-2 architecture at 75M, 125M, and 250M parameters. The 1.7 generation is a separate architecture experiment, and Boris 2, now in preview, is a new 143.7M-parameter architecture that builds on it. Both are described below.
Architecture
| Boris-1.3-75M | Boris-1.3-125M | |
|---|---|---|
| Architecture | GPT-2 — pre-LN, learned positional embeddings, tied embeddings | |
| Layers / heads / d_model | 12 / 9 / 576 | 12 / 12 / 768 |
| Parameters | 77.4M | 125M |
| Context length | 1024 | 1024 |
| Vocab | 50,304 (GPT-NeoX-20B BPE, padded) | |
| Tokenizer | EleutherAI/gpt-neox-20b | |
| Precision | bf16 autocast with fp32 master weights | |
1.0 and 1.3
The original Boris models (July 2026) were pretrained on FineWeb-Edu alone. Their benchmark results showed a consistent gap: strong enough on the knowledge-flavored tasks, weak on the fluency and coherence tasks — LAMBADA and WinoGrande in particular — which is what a purely educational-text diet produces.
Boris 1.3 is the same base checkpoints extended with continued pretraining passes that mix in DCLM-baseline, each pass with a re-warmed learning rate and each measured individually. The tradeoff discovered along the way is that DCLM improves LAMBADA and WinoGrande but tends to cost ARC-Easy and ARC-Challenge. At 125M, leading with FineWeb-Edu before introducing DCLM avoided that regression. Full pass-by-pass numbers are on the benchmarks page.
Training
| Run | Data | Tokens | Wall-clock (RTX 3060) |
|---|---|---|---|
| Boris-125M base | FineWeb-Edu | 2.50B | 33h 38m 48s |
| Boris-1.3-125M continued | FineWeb-Edu + DCLM-baseline, 7 passes | +2.66B | ~37h |
| Boris-75M base | FineWeb-Edu | 1.55B | — |
| Boris-1.3-75M continued | DCLM-baseline + FineWeb-Edu, 3 passes | +2.40B | — |
Boris-125M's final base-run figures were a loss of 3.2998, a gradient norm of 0.281, and a final learning rate of 6.00e-05. Pass-7 figures for the 1.3 continued run were not preserved and are reported as unavailable on the model card rather than estimated after the fact.
Instruction tuning
The Instruct variants are finetuned on
smol-smoltalk
and OpenAssistant (OASST1) conversational data. Boris-1.3-125M-Instruct took
5.6 hours on one RTX 3060. They ship a chat template, so
tokenizer.apply_chat_template() and llama.cpp's conversation
mode both work without extra configuration. See Usage.
Boris 1.7 — the n-gram table
Boris-1.7-D60M-n30M is a different question from 1.0 and 1.3, not a continuation of them. It puts a 60M-parameter dense trunk next to a 30M-parameter n-gram embedding table, spliced into the residual stream after block 2, and asks whether 30M of lookup buys what 30M of dense weight would at a fixed 2.5B-token budget. The trunk is deliberately conventional so that the table is the only thing under test.
| Boris-1.7-D60M-n30M | |
|---|---|
| Trunk | 10 layers, d=768, 12 heads / 4 KV heads (GQA), SwiGLU 1920, RoPE, RMSNorm, tied embeddings |
| Dense (non-embedding) params | 59,981,568 |
| Embedding params | 25,165,824 (tied) |
| N-gram table | 468,751 rows × rank 64, plus a 64→768 projection = 30,049,216 |
| Context / vocab | 1024 / 32,768 — its own byte-level BPE, trained on this corpus |
| Data | Ultra-FineWeb bulk, then cosmopedia-v2 + FineWeb-Edu anneal over the final 15% |
| Schedule | WSD (2% warmup, 85% stable, 1-sqrt decay); Muon on 2-D linear maps, AdamW elsewhere |
| Run | 2.5B tokens, 10,172 steps at 245,760 tokens/step, 22h 37m, bf16 + torch.compile |
| Final val | loss 3.235, ppl 25.4, 1.0012 bits/char |
The table is keyed on the literal last 2 and 3 token ids — 281,250
bigram rows, 187,500 trigram rows, and one shared out-of-vocabulary row. No
hashing: lookup is an exact searchsorted, one gather per
position and no matmul, so compute per token barely moves. Hit rates on the
finished model run about 65% bigram and 21% trigram.
Ablating the table at inference (--no-ngram) measures how
much the trained model leans on it. That is not the dense control —
that would need its own training run — but it is informative:
| Task | Metric | Intact | Table ablated | Δ |
|---|---|---|---|---|
| HellaSwag | acc_norm | 31.22 | 30.54 | −0.68 |
| ARC-Easy | acc_norm | 45.88 | 43.77 | −2.11 |
| ARC-Challenge | acc_norm | 26.02 | 22.95 | −3.07 |
| PIQA | acc_norm | 62.57 | 60.55 | −2.02 |
| WinoGrande | acc | 51.07 | 50.12 | −0.95 |
| LAMBADA | acc | 26.66 | 25.33 | −1.33 |
| Validation | bits/char | 1.0012 | 1.0699 | +0.0687 |
Removing the table costs 0.069 bits/char and between 0.7 and 3.1 points across the six tasks. The two ARC splits move most; HellaSwag and WinoGrande barely notice. Scored with the model's own tokenizer, on a harness first validated against GPT-Neo-125M's published numbers. It also scores 36.1 ±1.5 on ArithMark-3 (random is 25) and 1022 Elo on BananaMind Base Bench 1.1.
Because 1.7 carries a custom architecture and its own tokenizer, it is not
a transformers checkpoint — load it with the loader in its
repository. Checkpoints are resumable to the byte: optimizer states, data
cursor, and every RNG round-trip through the checkpoint.
Boris 2 — preview
Boris-2 is a 143.7M-parameter base model trained from scratch on a single NVIDIA V100 32GB. The main run continues to 200B tokens; the previews are checkpoints from it with a short, 2.5B-token learning-rate decay applied — Boris-2-Preview-1 at 50B tokens and Boris-2-Preview-2 at 100B. Both are base models: they continue text and do not follow instructions.
| Boris-2 | |
|---|---|
| Layers / heads / d_model | 31 / 8 × 64, full multi-head / 512 |
| MLP | SwiGLU, inner 1664 |
| Position encoding | RoPE, theta 10,000 |
| Context / vocab | 2048 / 32,768 — its own byte-level BPE |
| Parameters | 143.7M: 111.9M transformer blocks, 16.8M token embedding (tied), 15.0M n-gram table |
Two components differ from a standard decoder. Canon layers: each block runs a causal 3-tap depthwise convolution over the fused query/key/value projection, so every position mixes in the two before it ahead of attention. An n-gram table, the idea Boris-1.7 tested: an exact lookup on the last two and three token ids (210,938 bigrams and 23,437 trigrams, no hashing) retrieves a learned rank-64 vector that is added to the residual stream before block 2. It is a gather, not a matmul. Training packs several documents per sequence with attention, Canon and the n-gram lookup all masked at document boundaries, and the released modeling code does the same.
Zero-shot results, using lm-evaluation-harness 0.4.12 for the first four
and the Open SLM Leaderboard's unmodified bencharithmark-3.py
for ArithMark-3. The Open SLM Index is the leaderboard's chance-normalized,
weighted combination of the five.
| Task | Metric | Preview-1 | Preview-2 |
|---|---|---|---|
| HellaSwag | acc_norm | 45.23 | 46.60 |
| ARC-Easy | acc_norm | 58.38 | 59.13 |
| ARC-Challenge | acc_norm | 29.95 | 30.20 |
| PIQA | acc_norm | 69.21 | 70.84 |
| ArithMark-3 | acc_norm | 37.60 | 42.20 |
| Open SLM Index | 27.91 | 30.58 |
Single-checkpoint scores move by about ±0.8 Index between neighbouring checkpoints, and no checkpoint or seed was selected on them: each preview is the one checkpoint at the end of a decay fixed in advance. Preview-1's math data did not transfer to ArithMark-3's question format. Preview-2's gain there comes from grade-school word-problem data added to its decay, and so far it is specific to ArithMark: on SVAMP (35.3) and ASDiv (39.3), used as held-out checks, it is level with the pilot baseline.
For context, against the leaderboard's published figures for other models:
| Model | Params | Tokens | Index |
|---|---|---|---|
| Boris-2-Preview-2 | 144M | 102.5B | 30.58 |
| Boris-2-Preview-1 | 144M | 52.5B | 27.91 |
| cagliostro-v3.5 | 146M | 75.4B | 27.49 |
| SmolLM2-135M | 135M | 2T | 27.13 |
| cagliostro-v3 | 146M | 75B | 26.55 |
| SmolLM-135M | 135M | 600B | 25.74 |
| GPT-X2.5-135M | 135M | 75B | 25.17 |
| Training | Boris-2 |
|---|---|
| Tokens | 52.5B (Preview-1) and 102.5B (Preview-2), each ending in a 2.5B-token decay |
| Hardware | 1× NVIDIA V100 32GB |
| Precision | float16 mixed precision, float32 residual stream |
| Batch | 524,288 tokens per step, sequence length 2048 |
| Optimizer | Muon on matrices (lr 0.0025), AdamW on embedding, n-gram table, norms and Canon (lr 0.003) |
| Schedule | warmup, constant learning rate, then 1-sqrt decay to zero |
| Throughput | about 59,000 tokens/s, with hand-written Volta attention and fused kernels |
| Data | Main run about 89% ClimbMix and 11% FineMath, with UltraData-Math replacing FineMath from 55B tokens; decays add Cosmopedia WikiHow-style tutorials, math QA and a little DCLM-Baseline |
No benchmark training, validation or test split was trained on. Every source except ClimbMix — the bulk of the data, which is unfiltered — was checked against every evaluation item with a 13-gram overlap filter. Because the Cosmopedia tutorials were generated from WikiHow titles and about two thirds of HellaSwag comes from WikiHow, the model cards split HellaSwag by whether an item's title appears in the tutorial data. For Preview-2 both groups gained the same (+7.22 and +7.04 points), so the gain is general how-to fluency rather than topic overlap.
The previews load through transformers with
trust_remote_code=True; the custom code is plain PyTorch with
no KV cache. Run in float32 or bfloat16, not float16. See
Usage.
Limitations
A model of this size produces text that is frequently inaccurate, inconsistent, or offensive. Boris has received no alignment or safety tuning. Do not use it as a factual reference and do not deploy it without supervision.