Measured on one GPU. Reported plainly.
Zero-shot lm-evaluation-harness, full test sets, one run, same machine. Heimdall-Bit 337M against GPT-2, Pythia and SmolLM2.
| Test | Heimdall-Bit 337M | GPT-2 124M | Pythia-160M | Pythia-410M | SmolLM2-135M |
|---|---|---|---|---|---|
| Training tokens | 3.7B | WebText | ~300B | ~300B | ~2T |
| ARC-Easy (acc) | 45.0 | 43.8 | 43.2 | 51.9 | 64.3 |
| ARC-Challenge (acc_norm) | 23.9 | 22.7 | 23.1 | 24.2 | 29.7 |
| HellaSwag (acc_norm) | 32.0 | 31.1 | 30.2 | 40.5 | 43.2 |
| PIQA (acc) | 60.8 | 62.9 | 62.1 | 66.7 | 68.3 |
| Winogrande (acc) | 49.2 | 51.6 | 52.6 | 53.8 | 53.4 |
| LAMBADA (acc / ppl)† | 22.5 / 84.8 | 32.6 / 40.1 | 32.7 / 38.2 | 51.5 / 10.8 | 43.0 / 19.1 |
| WikiText word-perplexity ↓ | 41.7 | 37.4 | 33.5 | 20.1 | 23.1 |
Standard error is about ±1 point (±1.4 on Winogrande). Heimdall was evaluated with our adapter (bench/lmeval_heimdall.py, 2,048-token windows); the public models through the harness’s Hugging Face backend (GPT-2 with 1,024-token windows, Pythia and SmolLM2 with 2,048). † LAMBADA accuracy and perplexity depend on the tokenizer (our 8K vocabulary splits words into more tokens) and are not comparable across models; WikiText word-perplexity is.
What this does, and does not say.
What it says
With about 1/80 of Pythia’s training tokens, Heimdall-Bit is in the same range as GPT-2 small and Pythia-160M on ARC-Easy, ARC-Challenge and HellaSwag (differences within about a point). That is a model trained in about 92 hours on a desktop GPU.
What it does not
It is behind on PIQA (1 to 2 points), at chance on Winogrande and ARC-Challenge, behind on LAMBADA and WikiText (41.7 vs 33 to 37), and clearly behind Pythia-410M and SmolLM2-135M. It has more than twice the parameters of the models it matches, and with no dense baseline we cannot attribute any of this to the 1-bit FFN. Our mixture also spends 25 % of tokens on code, which these tests do not reward.
Versus our earlier research model
On 200 held-out windows of 512 tokens from a GitHub code shard neither model trained on, Heimdall-Bit 337M scores 1.011 nats against 3.226 for our earlier 116M model (positions 256 to 512: 0.874 vs 3.249). The earlier model was trained mostly by repeating a small corpus, so this shows the new model generalizes far better. It is not an equal-size or equal-data comparison of architectures.
Reproduce every number
The harness adapter, the prompts, the seeds and the scripts for each table are in the repository. github.com/valeroisai/heimdall
Go on. Try it.
Nova is free to start. No card, no setup afternoon. Or talk to us about the research.