Introducing Nandi-Mini-150M

Overview

A Foundation Model for Small Hardware

Nandi-Mini-150M is a decoder-only Transformer trained from scratch, with no borrowed weights. It is the base of the Nandi-Mini family.

Head dimension is hidden size divided by head count. Source: the Nandi-Mini-150M model card and its config.

Architecture

The Architecture at a Glance

Here is the whole model in one picture, read from the bottom up: tokens enter at the bottom, pass through the shared blocks, and leave as next-token probabilities at the top. The two side panels zoom in on the parts that make Nandi-Mini different.

Depth

Sixteen Layers, Thirty-Two Deep

Depth is what lets a Transformer refine its understanding: early layers settle which words are present, later layers settle what they mean together. But depth is paid for in parameters, and at 150 million there is not much to spend.

Nandi-Mini spends the same weights twice. The model stores 16 decoder layers. Each one is applied two times in a row before the signal moves on to the next, so every token passes through 32 layers of computation while the model stores only 16 layers of weights.

One way to picture it: the first pass writes a rough reading of each token in context, and the second pass, using the very same weights, reads that draft and sharpens it. There are no extra per-repeat parameters. The second pass uses the same attention, the same feed-forward and the same norms.

By that arithmetic, a conventional 32-layer model of this width would carry about 281 million parameters. Nandi-Mini carries about 153 million. That is the difference between a 560 MB download and a 307 MB one.

Embeddings

A Big Vocabulary Without the Weight

A vocabulary of 131,072 tokens is generous. It is how Nandi-Mini fits ten Indic scripts without shredding them, as the language results below show. But a vocabulary has a cost. A plain embedding table at the model's width of 832 would hold 131,072 × 832 parameters, about 109 million, nearly as many as the entire 16-layer stack at about 127 million.

So the table is factored. Tokens are looked up in a narrow table of width 196, and a small linear layer lifts them to 832 to enter the blocks. On the way out the model does the reverse: a projection down to 196, then the same table read backwards to produce the logits. Input and output share one table.

The table drops from about 109M parameters to about 26M, and the two projections add 0.3M. That returns roughly 83M parameters to the part of the model that thinks.

Block

The Rest of the Block

Around those two ideas sits a deliberately conventional core, chosen because it is well understood and fast on common hardware.

Languages

Built for Indic Scripts From the First Token

A tokenizer decides how much of your context window a sentence uses. A vocabulary trained mostly on English breaks Indic scripts into many small fragments, so the same Telugu word that costs one or two tokens in a purpose-built vocabulary can cost a dozen or more elsewhere. Every extra token is slower, costlier and takes room from a context window that, at 2,048 tokens, is already short.

Nandi-Mini's tokenizer was built for English and ten Indic languages together. Measured as tokens per word, Qwen3-0.6B-Base needs between 3.9× (Hindi) and 7.6× (Telugu) as many tokens as Nandi-Mini across the Indic languages we measured.

In practical terms, 2,048 tokens hold about 153 Telugu words under Qwen3's tokenizer and about 1,157 under Nandi-Mini's.

Against Sarvam-1, a tokenizer also built for Indic languages, Nandi-Mini is ahead on nine of twelve languages including English, level on Marathi, and behind on Hindi (1.32 against 1.25) and Odia (2.68 against 2.18). Assamese, at 1.51, is the widest gap in our favor.

Tokens per word, lower is better. Best in each row in bold. Source: the Nandi-Mini-150M model card.

Results

What 500 Billion Tokens Buy

We compared Nandi-Mini-150M with the models closest to it in size, using the same lm-eval setup for every row. Mobile-LLM figures come from its paper, because those checkpoints are not public, so it appears only in the full table. GSM8K and HumanEval use greedy decoding. No benchmark-specific tricks went into these numbers.

Nandi-Mini-150M leads its class on average (25.63, against 24.63 and 24.05) and on GPQA, MMLU, GSM8K and HumanEval, using 500B tokens where SmolLM2-135M saw 2 trillion. The commonsense tasks go the other way: SmolLM-135M and SmolLM2-135M score about 5 points higher on HellaSwag. And at this size math and code are barely awake, with GSM8K at 2.58 and HumanEval at 4.27. These are base-model numbers, a starting point for fine-tuning rather than a finished assistant.

Fine-Tuned for Hindi

A base model earns its keep when you adapt it. We fine-tuned each base model on CrossSum-Hindi, a summarization task from IndicGenBench, and scored it with chrF. Nandi-Mini-150M comes out on top, ahead of two models more than three times its size.

For Perspective

Larger models trained on far more data score higher on general benchmarks, and we show them here for the same reason we show the rest: so you can choose with the whole picture.

Family

The Nandi-Mini Family

Nandi-Mini-150M is the base of a growing family. Every model is on Hugging Face with open weights.

The whole collection lives in the Nandi-Mini collection.

Run

Run It Yourself

The architecture is custom, so loading it needs trust_remote_code=True. Everything else is standard Hugging Face.

# pip install transformers==5.4.0
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "FrontiersMind/Nandi-Mini-150M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, trust_remote_code=True, dtype=torch.bfloat16
).to(device).eval()

inputs = tokenizer(["The night was quiet and the streets were empty."],
                   return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True,
                         temperature=0.3, top_k=20, top_p=0.95,
                         repetition_penalty=1.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For serving, Nandi-Mini-150M runs on vLLM and SGLang with OpenAI-compatible endpoints, and on Docker Model Runner with one command:

docker model run hf.co/FrontiersMind/Nandi-Mini-150M

The exact serving commands are on the model card.

Fit

Where Nandi-Mini Fits

Reach for Nandi-Mini-150M when the constraint is the hardware: an edge device, an on-premises server, a latency budget measured in milliseconds, or a workload in an Indic language where a larger model's tokenizer wastes most of your context. It is a strong place to start a fine-tune, because the base is small enough to adapt quickly and cheaply, and the data and the weights never have to leave your infrastructure.

Know its edges. It is a base model, so it continues text rather than following instructions; use the Instruct or Tool-Calling releases, or fine-tune. Its context window is 2,048 tokens. Its math and code scores are low, as expected at 150 million parameters. And on general benchmarks, models many times its size trained on far more data are ahead.

The Takeaway

Nandi-Mini-150M is a small model with a design behind it: layers that run twice, an embedding table that does double duty, and a tokenizer that treats Indic languages as first-class. It was trained from scratch, it is open, and you can run it on hardware you own. That is where Sovereign Superintelligence starts.