Introducing Lumma-0.6B

Overview

A Foundation Model for Long Context

Lumma-0.6B is the 0.6-billion-parameter base model of the Lumma family. It was trained from scratch, with no borrowed weights, and it reads up to 12,288 tokens at once.

Head dimension is hidden size divided by head count. Source: the Lumma-0.6B-Base model card and its config.

Architecture

The Architecture at a Glance

Here is the whole model in one picture, read from the bottom up: tokens enter at the bottom, pass through 30 blocks, and leave as next-token probabilities at the top. The panel on the right zooms in on the attention layer, where Lumma differs most.

Shared KV

One Tensor, Two Jobs

Standard attention keeps two things for every token it has read: a key, which says what the token is about, and a value, which says what it contributes. Grouped-query attention shrinks both by sharing them across heads, but it still stores both.

Lumma-0.6B has no separate value projection at all. Each token is projected once, into a single latent. From that latent the model reads two things. The value is the latent exactly as stored. The key is the same latent after a light normalization and a rotary position encoding. One tensor goes into the cache, and two roles come out of it.

Keys and values are produced from the same token, so they are not independent, and letting them share a source costs little. The benchmark results below suggest that at this scale it does cost little, while halving what has to be remembered.

Position is the subtle part. RoPE rotates a key by an amount that depends on where its token sits in the sequence, so the cache holds the latent before rotation, and the model applies the key norm and the rotation to the history as it reads it. Values are never rotated.

That is the trade: memory for arithmetic. Reading the cache costs a little more compute per step, and in exchange the cache is half the size. For long context, where decoding is limited by how much memory has to be moved, that is the right side of the trade.

Memory

What Half a Cache Looks Like

The arithmetic is short. Each layer caches 8 key-value heads of width 90, which is 720 values per token. In vanilla mode that is stored twice, once for keys and once for values. Across 30 layers in bf16, that comes to 86.4 KB per token in vanilla mode and 43.2 KB in shared mode.

At the full 12,288-token window, one sequence needs about 1.06 GB of cache in vanilla mode and about 0.53 GB in shared mode. Serve 32 full-length conversations at once and the cache is 34 GB against 17 GB. That is memory you can spend on more users instead.

Block

The Rest of the Block

Around the shared cache sits a modern, stable core, with a few choices made for long windows and a small budget.

Languages

A Long Window Only Helps If Your Language Fits

Context length is counted in tokens, not words. A tokenizer that splits Indic scripts into many fragments quietly shrinks the window: 12,288 tokens is a long document in English and a short one in Telugu. Lumma-0.6B uses the same Indic-first tokenizer as Nandi-Mini, with a 131,072-token vocabulary built for English and ten Indic languages together.

The chart shows what that means for the window. In English the tokenizers are level. In Indic languages, a 12,288-token window holds between about 4 and 8 times as many words as it does under Qwen3-0.6B's tokenizer.

The tokenizer does not win everywhere. Sarvam-1 needs fewer tokens per word on Hindi (1.25 against 1.32) and on Odia (2.18 against 2.68). On Bengali, Gujarati, Kannada, Malayalam, Punjabi, Tamil, Telugu and Assamese, Lumma-0.6B's tokenizer is ahead of Sarvam-1, and on Assamese the gap is large: 1.51 tokens per word against 4.31.

Results

One Trillion Tokens, Measured Honestly

We evaluated Lumma-0.6B-Base on five standard tasks, next to the half-billion-parameter models closest to it. The most important column is not accuracy but tokens: Lumma-0.6B saw 1 trillion, and its comparisons saw between 1.3 and 18.

Read it plainly. Lumma-0.6B beats MobiLlama-0.5B on every task with fewer tokens. Against Qwen-2-0.5B it leads on the two ARC tasks and trails on HellaSwag, Winogrande and OpenBookQA, which nets out to a slightly higher average on one twelfth of the data. Against Qwen2.5-0.5B, trained on eighteen times as much, it is behind on all five. This is a base model, and the point of these numbers is that a strong foundation does not need the largest token budget.

Family

The Lumma Family

Lumma-0.6B-Base is the foundation of the 0.6B family, with open weights on Hugging Face.

The full set is in the Lumma collection.

Run

Run It Yourself

The architecture is custom, so loading it needs trust_remote_code=True. The cache mode is a one-line setting.

# pip install transformers==5.4.0
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "FrontiersMind/Lumma-0.6B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, trust_remote_code=True, dtype=torch.bfloat16
).eval()

# Memory-efficient mode (default): one cached latent per token
model.config.kv_cache_mode = "shared"
# Standard mode: keys and values cached separately
# model.config.kv_cache_mode = "vanilla"

inputs = tokenizer("The world is a strange place", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True,
                         temperature=0.3, top_p=0.95, top_k=20,
                         repetition_penalty=1.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For serving, Lumma-0.6B-Base runs on vLLM and SGLang with OpenAI-compatible endpoints, and on Docker Model Runner with one command:

docker model run hf.co/FrontiersMind/Lumma-0.6B-Base

The exact serving commands are on the model card.

Fit

Where Lumma Fits

Reach for Lumma-0.6B when context and memory are the constraint: long documents in English or Indic languages, many concurrent sessions on limited accelerators, or on-premises serving where every gigabyte of cache is a gigabyte you own and pay for. It is a good base for fine-tuning, and the instruct, tool and extraction releases are there when you want a starting point that already follows directions.

Know its edges. It is a base model, and base models are not recommended for conversation without post-training: supervised fine-tuning, RLHF or continued pre-training. On general benchmarks it sits behind Qwen2.5-0.5B, which saw far more data. And its tokenizer trails Sarvam-1 on Hindi and Odia.

The Takeaway

Lumma-0.6B asks a simple question of the KV cache: why remember two things when one will do? Keys and values read from a single stored latent, a cache half the size, and a window that stays useful in the languages it was built for. It was trained from scratch, it is open, and it runs on hardware you own. That is Sovereign Superintelligence in a small package.