littlerock

littlerock-1M is a decoder-only language model with 1,002,720 parameters — 609,504 of them outside the embedding table — trained from scratch on 5 billion tokens. It exists to measure what capability actually survives at a one-million-parameter budget.

Architecture

Property	Value
Architecture	Llama (loads with stock Transformers, no custom code)
Parameters	1,002,720 (609,504 non-embedding)
Layers	6
Hidden dimension	96
Attention heads	6 (16 dims per head)
FFN hidden size	224 (SwiGLU)
Context window	512
Vocabulary	4,096
Position encoding	RoPE
Normalization	RMSNorm, tied embeddings

Training

5,000,000,000 tokens from a mixture of 40% FineWeb-Edu and 60% DCLM-baseline, across 19,073 steps at 262,144 tokens per step, with AdamW and a cosine learning-rate schedule. Total wall-clock: 1 hour 49 minutes on one RTX 3060.

It is a next-token predictor with no instruction tuning and no alignment.

littlerock-1M-arithmax: a negative result worth publishing

littlerock-1M-arithmax is littlerock-1M finetuned on synthetic arithmetic word problems, generated from templates written for this project. It is published as a research artifact, not as a model to use.

ArithMark-3 set	Score
Public (published set)	40.4%	strong for 1M parameters
Private held-out (memorization test)	35.5%	passes — above the 25% floor
Private variety (rephrased questions)	21.5%	fails — below the floor
Base model, before finetuning	25.2%	at chance

Scores on the two private sets were measured independently by AxiomicLabs. On the variety-set result the finetune was declined for leaderboard ranking, correctly, and the base model was listed instead.

Dropping below chance is the interesting part. A model that had learned nothing would score 25%. Scoring 21.5% means the finetune taught surface cues strong enough that unfamiliar phrasing actively pulls the model toward wrong answers.

The protocol was sound and still missed it

No ArithMark item was trained on. The benchmark has no train split; all arithmetic training data was generated from project-written templates.
Every training string was checked against an index of all 16,428 items across all five scoring splits — exact context, exact context+ending, and 13-gram overlap. In control tests the guard caught 200/200 real ArithMark items, 200/200 HellaSwag, 200/200 PIQA, and 100/100 case- and punctuation-edited variants, and it dropped 221 genuine collisions from the corpus.
Checkpoints were selected on a held-out dev set, never on the benchmark.
The arithmetic dev set held out entire template families, not just instances, specifically to detect memorization. It reported 36.9%.

The independent variety set reported 21.5% — a ~15-point error in the project's own estimate of its own generalization. The cause is structural: the same author wrote both the training templates and the held-out ones, so both shared number ranges, unit phrasing, and sentence rhythm. A held-out split cannot detect distribution-level specialization when you wrote both sides of it. An earlier iteration was worse still: 87.9% against a same-template dev set versus 26.9% on the real benchmark, a 61-point gap.

Per-topic breakdown (public set)

Topic	Score	Covered by the generator?
division	72.2	yes
multiplication	56.8	yes
subtraction	48.7	yes
addition	39.8	yes
two-step add/subtract	15.2	yes, did not transfer
fractions: counting	24.0	no
patterns	13.2	no

High on covered topics, at or below chance on uncovered ones — the opposite of what test-set contamination produces. That was taken as evidence the ability was real. It is good evidence the ability was not copied from the test set, which turns out to be a weaker claim than it appears.

Takeaway

Benchmark specialization can survive a rigorous contamination audit, honest dev-set selection, and a plausible-looking capability profile. Detecting it needs evaluation data the model's author did not write. That is an argument for private variety sets, and a caution against trusting held-out splits you built yourself.