Models that stay light as they get smarter.
Every Valerois model is built on Heimdall. One scratch-trained model is trained and evaluated today. What is planned is marked as planned.
Heimdall-Bit 337M
Hybrid · 1-bit FFN · base model
The largest model trained from scratch: 3.7B tokens of FineWeb-Edu (55 %), Cosmopedia (21 %) and GitHub code (25 %) on one AMD Radeon RX 9070 XT. A base model without instruction tuning. Weights are not released; the code is source-available for review.
Heimdall-Bit 98M
Study model · 800M tokens
The mid-scale model of the bit-versus-dense study. It trained for 4.7 hours at a steady 47.4K tokens per second with no crash. At this size the 1-bit FFN is slightly behind a dense model half its size.
Heimdall-converted SmolLM2-135M
Built on SmolLM2 by Hugging Face
A pretrained open Transformer turned into a constant-memory model by training about 2 % new parameters for 0.8M tokens. It is a result of the method, not a standalone Valerois model.
Scale study · 300M to 1B
Dense baseline and bit hybrid
Train the binary-FFN hybrid and a dense baseline side by side on at least 10B tokens with several seeds. This answers the main open question: whether the 1-bit gap closes at scale. It depends on compute; nothing here is a result yet.
The specs, to scale.
| Model | Status | Params | Tokens | Hardware | Cache |
|---|---|---|---|---|---|
| Heimdall-Bit 337M | trained | 337M | 3.7B | RX 9070 XT 16 GB | 12.7 MB + 11.8 KB/token |
| Heimdall-Bit 98M | trained | 98M | 800M | RX 9070 XT 16 GB | — |
| Converted SmolLM2-135M | conversion | 135M | 0.8M (adapt) | RX 9070 XT 16 GB | 5.7 MB constant |
| Scale study | planned | 300M – 1B | ≥ 10B | needs compute | — |
See the benchmarks Read the researchRequest evaluation access
Go on. Try it.
Nova is free to start. No card, no setup afternoon. Or talk to us about the research.