Transformer alternatives: constant-memory language models

The short answer

Mamba, RetNet, DeltaNet and Gated DeltaNet, RWKV and xLSTM are the best known alternatives to the Transformer. They keep a fixed-size memory instead of a cache that grows. The cost is exact recall of far-away details, which is why many recent designs, including Jamba, Griffin and our Heimdall, are hybrids that keep a few attention layers.

Why memory is the problem

When a Transformer writes the next word it attends to every word before it, and to avoid recomputing them it keeps their keys and values: the KV cache. The cache grows linearly with the context. In our conversion experiment, SmolLM2-135M needs 180 MB of cache at 8K tokens and about 22 GB at 1M tokens. On a laptop or a single consumer GPU, that cache, not the weights, is what limits how much text you can work with.

The families

Family	Idea	Memory as text grows	Key papers
Transformer (full attention)	Every token looks back at all earlier tokens.	Grows with every token	Vaswani et al., 2017
State space models: Mamba, Mamba-2	A fixed-size state updated by rules that depend on the input (“selective”).	Constant	Gu and Dao, 2023 · Dao and Gu, 2024
Retention and linear attention: RetNet	Attention rewritten so it can run as a recurrence over a matrix state.	Constant	Sun et al., 2023
Delta-rule models: DeltaNet, Gated DeltaNet	A matrix memory written with the delta rule, which corrects what is stored; the gated version can also forget.	Constant	Yang et al., 2024 · Yang, Kautz and Hatamizadeh, 2024
Modern recurrent networks: RWKV, xLSTM	Recurrent networks redesigned so they can be trained in parallel like Transformers.	Constant	Peng et al., 2023 · Beck et al., 2024
Hybrids: Griffin, Jamba, Heimdall	Mostly recurrent layers, with a few attention layers kept for exact recall.	Grows only in the attention layers	De et al., 2024 · Lieber et al., 2024

Descriptions are simplified. The links go to the original papers on arXiv.

What you give up

A memory of fixed size has to decide what to keep. Tasks that need an exact detail from far back, such as retrieving a specific value planted earlier in the text, are where pure recurrent models struggle most, and where a few attention layers help. In our own toy associative-recall test (about 100K parameters, one seed, so small and noisy), attention alone fell from 15.6 % to 0.8 % as the gap grew, pure DeltaNet reached about 14.8 %, and the hybrid scored 99.9 to 100 %. That is consistent with the hybrid design, and it is not evidence at scale.

Where Heimdall fits

Heimdall is a hybrid: for every global-attention layer there are three Gated DeltaNet layers, so only one layer in four keeps a growing cache. We measured the 337M model at 18.7 MB of cache at 512 tokens and 109 MB at 8,192 tokens, about 11.8 KB per token, roughly 12 times smaller than a same-size Transformer by calculation. A second part of Heimdall converts a pretrained Transformer (SmolLM2-135M) into a constant-memory model with 0.8M training tokens: a 5.7 MB cache at any length, with WikiText perplexity going from 14.09 to 17.61.

We are careful about what this shows. Heimdall-Bit scores at roughly GPT-2-small and Pythia-160M level and is behind SmolLM2-135M and Pythia-410M. There is no dense baseline at this scale, so we do not know whether the hybrid layout or the 1-bit layers help or hurt the scores. The conversion is not at parity (+25 % perplexity) and long-context retrieval is not established. Every limitation is on the research page.

1-bit weights are a separate axis

Making the weights smaller is a different idea from making the memory constant. BitNet (Wang et al., 2023) and BitNet b1.58 (Ma et al., 2024) train models with 1-bit and ternary weights. Heimdall combines both: a hybrid backbone for the cache, and feed-forward layers trained with binary or ternary weights directly, with no full-precision copy of the weights, using a custom optimizer called BitFlip. At about 100M parameters the 1-bit model is slightly behind a half-size dense model (+0.02 nats), so we do not claim it pays off at scale.

Read the Heimdall report See the benchmarks Glossary

Questions

What is an alternative to the Transformer?

Any language model architecture that replaces most of full attention with something whose memory does not grow with the text. The best known are state space models such as Mamba, RetNet, delta-rule models such as DeltaNet and Gated DeltaNet, RWKV, xLSTM, and hybrids such as Griffin, Jamba and Heimdall.

Why does the Transformer use so much memory?

Attention keeps the keys and values of every earlier token (the KV cache) so it does not recompute them. That cache grows linearly with the context. For SmolLM2-135M it is 180 MB at 8K tokens and about 22 GB at 1M tokens.

What do constant-memory models give up?

A fixed-size memory cannot hold everything, so exact recall of details far back in the text is the hard part. Hybrids keep a few attention layers for that.

Is Heimdall better than a Transformer?

We do not claim that. At 337M parameters it scores at roughly GPT-2-small and Pythia-160M level, and there is no dense Transformer baseline at this scale yet, so the effect of the hybrid layout on scores is unknown. What we measured is the memory: about 12 times smaller cache than a same-size Transformer, by calculation.

Can a language model be trained on one consumer GPU?

A small one, yes. Heimdall-Bit 337M was trained from scratch on 3.7B tokens in about 92 hours on one 16 GB AMD Radeon RX 9070 XT. It is a base model, not an assistant.

References

Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762.
Gu and Dao (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
Dao and Gu (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. arXiv:2405.21060.
Peng et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. arXiv:2305.13048.
Sun et al. (2023). Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621.
Yang et al. (2024). Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484.
Yang, Kautz and Hatamizadeh (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464.
Beck et al. (2024). xLSTM: Extended Long Short-Term Memory. arXiv:2405.04517.
De et al. (2024). Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. arXiv:2402.19427.
Lieber et al. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv:2403.19887.
Wang et al. (2023). BitNet: Scaling 1-bit Transformers for Large Language Models. arXiv:2310.11453.
Ma et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764.
Valerois (2026). Heimdall: a constant-memory hybrid, trained on one GPU. github.com/valeroisai/heimdall.