Heimdall: a constant-memory hybrid, trained on one GPU.

Abstract

Heimdall combines three ideas. First, a hybrid backbone: three Gated DeltaNet layers, which keep a fixed-size recurrent memory, for every one global-attention layer, so the inference cache is about 12× smaller than a same-size Transformer’s (by calculation). Second, a latent-free 1-bit and 1.58-bit training engine: binary (±1) or ternary weights are the only copy of the weights, with no fp32 shadow weights and no Adam state, and a custom optimizer called BitFlip flips the strongest-gradient weights directly. Third, a conversion method that turns a pretrained open Transformer into a constant-memory model by training only about 2 % new parameters.

Everything was measured on one workstation: an AMD Radeon RX 9070 XT (16 GB) with a Ryzen 7 7800X3D. The largest model trained from scratch, Heimdall-Bit, has 337M parameters and saw 3.7B tokens. Unless stated otherwise, comparisons are single runs, so differences of about 0.03 nats or less are noise.

Architecture

Every fourth layer is global attention; the other three are Gated DeltaNet. A DeltaNet layer reads and writes a fixed-size matrix memory with the delta rule, so its state does not grow with the sequence. Only the attention layers keep a key-value cache.

Measured on the 337M model: the cache is 18.7 MB at 512 tokens and 109 MB at 8,192 tokens, growing about 11.8 KB per token because only the attention layers grow. The DeltaNet layers are constant. On a toy associative-recall task (MQAR, about 100K parameters, 8 CPU-minutes, one seed) trained at gaps up to 64 tokens and tested up to 16K, attention-only fell from 15.6 % to 0.8 %, pure DeltaNet reached about 14.8 %, and the hybrid scored 99.9 to 100 %. Small and noisy, but consistent with the design.

Latent-free 1-bit training

Question tested: can a model whose feed-forward weights are 1-bit match a dense model with half as many parameters? Validation loss in nats per token at equal data, lower is better.

At small scale the bit model is clearly ahead; around 50M it ties; at about 100M it is slightly behind (+0.02 nats). We do not claim the target is met at scale. Binary and ternary FFN weights were equally good in these tests.

Training throughput on one 16 GB GPU

torch.compile on the ROCm 10.1 and PyTorch 2.14 stack gave 2.9× faster training and 2.4× lower memory than eager mode on a 5.8M model (90K to 264K tok/s, 3.1 to 1.3 GB) with unchanged validation loss.

Design study: where to put the bits

6M scale, 10M tokens. Most of the quality loss comes from binarizing the attention and DeltaNet projections. Keeping those dense and adding a small dense low-rank corrector next to each bit matrix recovers most of it. In the final design 58 % of parameters are 1-bit.

Converting a pretrained Transformer

SmolLM2-135M (30 layers), sliding window of 64 tokens plus 4 attention-sink tokens plus a delta-rule memory, 0.8M training tokens (about 4 minutes on the GPU). WikiText perplexity at 2K context:

About 70 % of the gap between window-only and the original is closed, with a 5.7 MB cache at any length (the original needs 180 MB at 8K tokens and about 22 GB at 1M). It is not at parity (+25 % perplexity) and long-context retrieval has not been established. 4-bit teacher weights cost +7 % perplexity on Qwen3-0.6B at 318 MB. Our reimplementation of the teacher matches Hugging Face transformers logits exactly on SmolLM2-135M and Qwen3-0.6B.

Kernels

Triton kernels for int8 and packed 1/2-bit matrix products. Single-token decoding, 4096×4096, RX 9070 XT, ROCm 10.1:

For prefill and training shapes our kernels are slower than bf16 (0.33 to 0.72×), so training uses plain bf16. These are layer benchmarks, not end-to-end generation speed.

Results

Final validation loss of the 337M run, in nats per token with our 8K-vocabulary tokenizer (not comparable to other models’ perplexities): FineWeb-Edu 2.707 and 2.668 (two splits), GitHub code 0.743 and 0.606, Cosmopedia 1.773. The code losses are optimistic because of boilerplate and near-duplicates. Samples are fluent English and plausible Python structure, with wrong facts, wrong logic and loops under greedy decoding: it is a base model with no instruction tuning.

Beyond the training length

Trained at 2,048 tokens. Loss keeps falling to about 4K, then rises. Passkey retrieval (a 5-digit number planted in filler text, 15 trials per length): 100 % at 512, 1K, 2K and 4K, and 80 % at 8K, where a far-placed key fails (2 of 5). Small samples on an easy test.

Standard benchmarks against GPT-2, Pythia and SmolLM2

Limitations

Next steps

1. Scale study: train the binary-FFN hybrid and a dense baseline at 300M to 1B parameters on at least 10B tokens, several seeds, to learn whether the bit-vs-dense gap closes at scale.
2. Evaluate converted models on standard benchmarks and long-context retrieval (RULER-style).
3. Convert larger open models (Qwen3 1.7B to 8B, 4-bit teacher) with the Heimdall conversion and measure long-context quality and serving cost.
4. Fused low-bit training kernels and end-to-end inference throughput measurements.

These depend on compute. If you can help, talk to us.

Reproduce

License: source-available for review only (read, evaluate, and run unmodified to verify the reported results). No usage rights are granted at this time. Code comments, CLI flags and some log messages are in Turkish; the repository README has a glossary.

Open the repository