How We Train

This page describes the actual procedure behind every OpenCerebral release, so that the model cards read as records rather than claims.

Hardware

One NVIDIA RTX 3060, 12 GB. Every model published to date — 1M through 250M parameters, text and image alike — was trained on that single card. Runs are measured in hours to days, not GPU-months, and the wall-clock figure is published for each one.

That is the compute available, not a target. The recipes scale before the hardware does, and more GPU time is the single thing that would most change what this project can publish — see Get Involved. What will not change with more compute is the reporting: wall-clock, token counts, and the passes that regressed stay in the model cards at any size.

Data

All public, in varying mixtures. The Boris 1.0 and 1.3 generations use two corpora:

Boris 1.7 uses a different mix — Ultra-FineWeb in bulk, with cosmopedia-v2 and FineWeb-Edu annealed over the final 15% of the run — and its own 32,768-entry byte-level BPE trained on that corpus rather than the GPT-NeoX tokenizer. The Lucy image models train on CC3M and CelebA.

Instruction tuning uses smol-smoltalk and OpenAssistant OASST1. Earlier Boris 1.0 instruct models used Alpaca. Nothing else is in the mix, and there is no private data.

Continued pretraining

Rather than retraining from scratch when a benchmark gap shows up, we extend an existing base checkpoint with additional passes, each with a re-warmed learning rate, and evaluate after every pass. Boris-1.3-125M is seven such passes on top of Boris-125M; Boris-1.3-75M is three on top of Boris-75M. Because each pass is measured separately, the resulting table shows which data helped which benchmark, including the passes that made things worse.

Evaluation

Six standard tasks — HellaSwag, PIQA, WinoGrande, ARC-Easy, ARC-Challenge, LAMBADA — reported individually and as an unweighted Mean-6. No benchmark data enters training, and no checkpoint is selected on a benchmark score.

Where this went wrong once

The arithmetic finetune of littlerock-1M passed every contamination check we ran and still failed on rephrased problems, at a rate our own held-out dev set missed by about 15 points. The write-up is deliberately detailed, because the lesson generalizes: a held-out split cannot detect distribution-level specialization when the same person wrote both sides of it.

What gets published

For every model: weights, tokenizer, config, architecture table, token counts per pass, dataset identities, wall-clock times, final loss and gradient norm where preserved, the full benchmark table, and the limitations. Where a figure was not preserved, the card says so instead of reconstructing it.