Glossary: Heimdall, Gated DeltaNet, KV cache and more

1-bit and 1.58-bit weights: Weights restricted to two values (±1, binary) or three values (−1, 0, +1, ternary, about 1.58 bits). Heimdall uses them in the feed-forward layers, 58 % of its parameters.
BitFlip: Heimdall’s custom optimizer. It flips the weights with the strongest gradients directly instead of nudging a full-precision copy.
Constant memory: A model whose memory for the text it has read does not grow with the length of the text, because it keeps a fixed-size state instead of a growing cache.
Conversion: Heimdall’s method for turning a pretrained Transformer into a constant-memory model by training only about 2 % new parameters. We converted SmolLM2-135M with 0.8M tokens.
Delta rule: An update rule that writes a new value into a matrix memory by correcting what is already stored for that key, instead of only adding to it.
Gated DeltaNet: A layer that combines the delta rule with a learned gate that lets the memory forget. Introduced by Yang, Kautz and Hatamizadeh in 2024. It keeps a fixed-size state.
Heimdall: Valerois’s research architecture: a hybrid of Gated DeltaNet and global attention with latent-free 1-bit feed-forward layers, trained from scratch on one consumer GPU. It is also the name of the research codebase.
Heimdall-Bit: The models we trained from scratch with the Heimdall design: 98M parameters (800M tokens) and 337M parameters (3.7B tokens). Base models, with no instruction tuning.
Hybrid architecture: A model that mixes layer types, usually many recurrent or state-space layers and a few attention layers, to keep memory low while keeping exact recall. Heimdall uses three Gated DeltaNet layers for every attention layer.
KV cache: The keys and values a Transformer saves for every earlier token so it does not recompute them. It grows linearly with the context and is the main memory cost of long generations.
Latent-free training: Training where the binary or ternary weights are the only copy of the weights: no full-precision shadow weights and no Adam optimizer state.
MQAR: Multi-query associative recall: a test where the model must return the values for many keys planted earlier in the text. It probes how well a fixed-size memory holds on to details.
Nats per token: Validation loss measured with the natural logarithm; lower is better. Only comparable between models with the same tokenizer and data, which is why we use word perplexity to compare across models.
Nova: Valerois Studio Nova: the AI workspace by Valerois that plans, writes, runs and checks your work on your computer or in the browser. Not affiliated with other products or models called Nova.
Perplexity: The exponential of the loss. Roughly, how many words the model is hesitating between at each step; lower is better.
Supernova mode: A Nova mode for work that has to hold: it plans, builds in proven steps, verifies, then has a second model review the result. Included in Pro and Max.
Transformer: The neural network architecture built on attention, introduced in 2017. Each new token attends to all earlier tokens, so its cache grows with the text.

Glossary: plain words for the hard ones

Go on. Try it.