How We Train

This page describes the actual procedure behind every OpenCerebral release, so that the model cards read as records rather than claims.

Hardware

One NVIDIA RTX 3060. Every model published to date — 1M through 250M parameters — was trained on that single card. Runs are measured in hours to days, not GPU-months, and the wall-clock figure is published for each one.

Data

Two public corpora, in varying mixtures:

Instruction tuning uses smol-smoltalk and OpenAssistant OASST1. Earlier Boris 1.0 instruct models used Alpaca. Nothing else is in the mix, and there is no private data.

Continued pretraining

Rather than retraining from scratch when a benchmark gap shows up, we extend an existing base checkpoint with additional passes, each with a re-warmed learning rate, and evaluate after every pass. Boris-1.3-125M is seven such passes on top of Boris-125M; Boris-1.3-75M is three on top of Boris-75M. Because each pass is measured separately, the resulting table shows which data helped which benchmark, including the passes that made things worse.

Evaluation

Six standard tasks — HellaSwag, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, LAMBADA — reported individually and as an unweighted Mean-6. No benchmark data enters training, and no checkpoint is selected on a benchmark score.

Where this went wrong once

The arithmetic finetune of littlerock-1M passed every contamination check we ran and still failed on rephrased problems, at a rate our own held-out dev set missed by about 15 points. The write-up is deliberately detailed, because the lesson generalizes: a held-out split cannot detect distribution-level specialization when the same person wrote both sides of it.

What gets published

For every model: weights, tokenizer, config, architecture table, token counts per pass, dataset identities, wall-clock times, final loss and gradient norm where preserved, the full benchmark table, and the limitations. Where a figure was not preserved, the card says so instead of reconstructing it.