Introducing NandiBERT
Open encoders that are state of the art in Indic and among the best in English, with Bahasa Indonesia, Malay and Arabic. Two sizes: 150M with a 16K window and 450M with a 32K window.
Encoders do the quiet work behind search, classification, tagging and retrieval-augmented generation. Until now, a team building for English and Indic languages had to choose: an English encoder that is strong in English, or an Indic encoder that is strong in Indic. NandiBERT is built so that choice goes away.
We are releasing two sizes with open weights under Apache 2.0. NandiBERT-150M reads 16,384 tokens and NandiBERT-450M reads 32,768. Both share one 100K-token vocabulary across English, 11 Indic languages, Bahasa Indonesia, Malay and Arabic.
Until now, IndicBERT v2 was the state of the art for Indic encoders. NandiBERT takes that place: NandiBERT-450M beats IndicBERT v2 on all seven Indic tasks we report, and NandiBERT-150M beats it on average with 45% fewer parameters. In English, both models are among the best, ranking second on average against English-only encoders of their size, ahead of ModernBERT and Ettin. This post covers the design and the numbers.
Overview
One Encoder, Many Languages
NandiBERT is a family of bidirectional Transformer encoders trained with masked language modeling. Both sizes share the same design and the same tokenizer, so what you learn on one carries to the other.
149M and 449M parameters, open weights under Apache 2.0
16,384 tokens for 150M, 32,768 for 450M
100,352 tokens, shared across the family
in the pre-training corpus
| Spec | NandiBERT-150M | NandiBERT-450M |
|---|---|---|
| Architecture | Bidirectional Transformer encoder | Bidirectional Transformer encoder |
| Parameters | 149M | 449M |
| Layers | 28 | 28 |
| Hidden size | 768 | 1,024 |
| Attention | Grouped-query: 6 query heads, 2 key-value heads, head dimension 128, QK-norm | Grouped-query: 8 query heads, 2 key-value heads, head dimension 128, QK-norm |
| Feed-forward | SwiGLU, intermediate size 1,280 | SwiGLU, intermediate size 3,712 |
| Embeddings | Factorized, rank 192 | Factorized, rank 512 |
| Vocabulary | 100,352 tokens | 100,352 tokens |
| Context | 16,384 tokens | 32,768 tokens |
| Positions | RoPE, base 1,000,000 | RoPE, base 1,000,000 |
| Objective | Masked language modeling, whole-word masking | Masked language modeling, whole-word masking |
| License | Apache 2.0 | Apache 2.0 |
Source: the NandiBERT-150M and NandiBERT-450M model cards and configs.
Architecture
Deep, Narrow and Built for Long Inputs
Both models are 28 layers deep. The 150M model is 768 wide and the 450M model is 1,024 wide, so most of the capacity goes into depth, where an encoder refines what each token means in context.
Grouped-Query, QK-Norm
Query heads share two key-value heads, which keeps attention memory low on long inputs. Queries and keys are normalized before attention.
SwiGLU
A gated feed-forward layer: 1,280 wide in the 150M model and 3,712 in the 450M model.
RoPE, Base One Million
Rotary positions with a base of 1,000,000, so positions stay distinct across 32,768 tokens.
A Big Vocabulary Without the Weight
A 100,352-token vocabulary is what lets one model cover many scripts without breaking them into fragments. Stored as a plain table, it would cost 77M parameters in the 150M model, half its budget. So the table is factored: tokens are looked up in a narrow table and a small projection lifts them to the model's width.
Figure 1 · Embedding Parameters, Full Against Factorized
That returns about 58M parameters to the encoder layers in NandiBERT-150M and about 51M in NandiBERT-450M.
Indic
State of the Art in Indic
Until now, IndicBERT v2 was the state of the art among Indic encoders. We compared both NandiBERT models with it on seven tasks: four for classification (IndicSentiment, IndicXNLI, IndicCOPA and IndicXParaphrase), two for structure prediction (Naamapadam NER and MASSIVE slot-filling) and one for question answering (IndicQA).
Figure 2 · NandiBERT-450M Against IndicBERT v2
NandiBERT-450M is ahead on all seven tasks and averages 73.2 against 65.9. The widest margins are on IndicXParaphrase (77.8 against 57.0) and IndicCOPA (76.1 against 63.0). That makes NandiBERT-450M the new state of the art for Indic encoders.
| Model | Params | IndicSentiment | IndicXNLI | IndicCOPA | IndicXParaphrase | Naamapadam NER | MASSIVE Slot-fill | IndicQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| IndicBERT v2 | 272M | 88.3 | 74.3 | 63 | 57 | 72.4 | 57.3 | 49.2 | 65.93 |
| NandiBERT-450M | 449M | 92.9 | 81.3 | 76.1 | 77.8 | 72.5 | 61 | 51 | 73.23 |
Figure 3 · NandiBERT-150M Against IndicBERT v2
NandiBERT-150M is ahead on five of seven tasks and on average (67.4 against 66.6), with 149M parameters to IndicBERT v2's 272M. IndicBERT v2 keeps the lead on MASSIVE slot-filling and IndicQA.
| Model | Params | IndicSentiment | IndicXNLI | IndicCOPA | IndicXParaphrase | Naamapadam NER | MASSIVE Slot-fill | IndicQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| IndicBERT v2 | 272M | 91.2 | 73.7 | 64.6 | 56.9 | 73.2 | 58.1 | 48.3 | 66.57 |
| NandiBERT-150M | 149M | 91.3 | 75.9 | 66.4 | 60.6 | 77.4 | 54 | 46 | 67.37 |
IndicBERT v2 scores as reported on each NandiBERT model card. Higher score in each column in bold.
English
Among the Best in English
We compared each NandiBERT model with English encoders of similar size on ten benchmarks: five for classification (GLUE, SQuAD 2.0, MultiCoNER v2, Few-NERD and RewardBench) and five for retrieval (BEIR, RAR-b Code, RAR-b Math, MLDR and LongEmbed). Every other model in these comparisons is English-only, so NandiBERT competes with specialists on their own ground.
Figure 4 · English Average, 150M Class
NandiBERT-150M averages 61.0, second to UltraBERT-150M and ahead of Ettin-Encoder-150M and ModernBERT-base. It ranks first on Few-NERD, and second on SQuAD 2.0, RewardBench and all five retrieval tasks.
| Model | Params | GLUE | SQuAD 2.0 | MultiCoNER v2 | Few-NERD | RewardBench | BEIR | RAR-b Code | RAR-b Math | MLDR | LongEmbed | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BERT-base | 110M | 84.7 | 76.3 | 53.8 | 66.6 | 59.4 | 36.7 | 6.6 | 27.5 | 29 | 29.3 | 46.99 |
| RoBERTa-base | 125M | 86.4 | 83.7 | 56.5 | 67.1 | 63.2 | 39.6 | 11.7 | 33.4 | 28.7 | 33.8 | 50.41 |
| MosaicBERT-base | 137M | 85.4 | 79.4 | 57.8 | 65.7 | 62.5 | 41.4 | 12.7 | 30.9 | 29.6 | 40.5 | 50.59 |
| NomicBERT-2048 | 137M | 84 | 82.7 | 60.1 | 66.9 | 64.1 | 42 | 13.8 | 30.2 | 29.9 | 45 | 51.87 |
| GTE-en-MLM-base | 137M | 85.6 | 82.7 | 59.2 | 66.4 | 63.4 | 42.4 | 12 | 33.6 | 37.1 | 58.5 | 54.09 |
| ModernBERT-base | 149M | 88.4 | 86.7 | 57.4 | 66.7 | 68 | 42.6 | 29.6 | 46.1 | 37.7 | 63.2 | 58.64 |
| Ettin-Encoder-150M | 149M | 88.9 | 87.4 | 57.4 | 66.5 | 70.3 | 42.4 | 31.8 | 47.1 | 35.9 | 60.1 | 58.78 |
| UltraBERT-150M | 152M | 89.5 | 89.2 | 57.9 | 67.4 | 73 | 45.2 | 39.1 | 55.4 | 43.5 | 72 | 63.22 |
| NandiBERT-150M | 149M | 87.1 | 87.6 | 56.9 | 68.1 | 71.2 | 44.5 | 32.8 | 49.4 | 41.2 | 71.1 | 60.99 |
Figure 5 · English Average, 400M Class
NandiBERT-450M averages 63.3, second to UltraBERT-380M and ahead of Ettin-Encoder-400M, LFM2.5-Encoder-350M and ModernBERT-large. It ranks first on Few-NERD. At 449M parameters it is also the largest model in this table.
| Model | Params | GLUE | SQuAD 2.0 | MultiCoNER v2 | Few-NERD | RewardBench | BEIR | RAR-b Code | RAR-b Math | MLDR | LongEmbed | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BERT-large | 334M | 85.2 | 81.8 | 57.7 | 67.8 | 62.6 | 38.6 | 12.2 | 33 | 29.8 | 30 | 49.87 |
| RoBERTa-large | 355M | 88.9 | 89.4 | 62.3 | 68.1 | 68.5 | 42.8 | 20.3 | 37.8 | 29.5 | 32.2 | 53.98 |
| GTE-en-MLM-large | 434M | 87.6 | 87.2 | 63 | 67.1 | 62.2 | 39.4 | 13.5 | 32 | 40.8 | 57 | 54.98 |
| NeoBERT | 222M | 89 | 86.6 | 57.4 | 66.4 | 57.6 | 42.9 | 17.3 | 33.4 | 21.7 | 32.4 | 50.47 |
| LFM2.5-Encoder-350M | 355M | 89.2 | 87.2 | 62.7 | 67.9 | 68.5 | 47.3 | 38.6 | 50.2 | 44.5 | 71.7 | 62.78 |
| Ettin-Encoder-400M | 395M | 90.8 | 90.4 | 62.3 | 67.8 | 76.7 | 45.9 | 38.5 | 54 | 40.9 | 64.3 | 63.16 |
| ModernBERT-large | 395M | 90.4 | 89.8 | 60.9 | 67.6 | 72.8 | 45.2 | 34 | 46.1 | 39.9 | 63.5 | 61.02 |
| UltraBERT-380M | 380M | 91 | 90.9 | 63.5 | 68.3 | 78.8 | 47.7 | 47 | 61.7 | 44.1 | 74.9 | 66.79 |
| NandiBERT-450M | 449M | 89.9 | 89.7 | 61.4 | 68.5 | 74.7 | 46.1 | 37.4 | 52.8 | 41.46 | 71.4 | 63.34 |
UltraBERT leads the English average in both size classes, and it is English-only. NandiBERT carries Indic, Bahasa Indonesia, Malay and Arabic in the same weights, and still places second.
Long Context
Whole Documents in One Pass
A contract, a court judgment or a research paper rarely fits in 512 tokens. NandiBERT-150M reads 16,384 tokens and NandiBERT-450M reads 32,768, so a whole document can go through the encoder at once instead of being cut into chunks that lose their context.
71.1 and 71.4
Long-document retrieval for NandiBERT-150M and NandiBERT-450M. The 150M model is second of nine in its class.
41.2 and 41.5
Long-document retrieval for the two sizes. The 150M model is second of nine in its class.
Family
The NandiBERT Family
Both encoders are on Hugging Face with open weights.
NandiBERT-150M
149M parameters and a 16K window, for fast classification, tagging and reranking.
NandiBERT-450M
449M parameters and a 32K window, for long documents and the strongest Indic results.
Embeddings
EmbeddingNandi: Coming Soon
An encoder becomes a search engine once it learns to turn a whole passage into one vector. EmbeddingNandi is that next step: sentence-embedding models fine-tuned from NandiBERT-150M and NandiBERT-450M, for semantic search, retrieval-augmented generation, clustering and deduplication, across the same languages.
EmbeddingNandi-150M
Built on NandiBERT-150M, for fast multilingual retrieval.
EmbeddingNandi-450M
Built on NandiBERT-450M, for long-document retrieval.
Run
Run It Yourself
The architecture is custom, so loading it needs trust_remote_code=True. Everything else is standard Hugging Face.
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
model_id = "FrontiersMind/NandiBERT-150M" # or FrontiersMind/NandiBERT-450M
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
model_id, dtype=torch.bfloat16, trust_remote_code=True, device_map="cuda"
)
texts = [
f"The capital of France is {tokenizer.mask_token}.",
f"भारत की राजधानी {tokenizer.mask_token} है।",
]
for text in texts:
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
print(tokenizer.decode(logits[0, mask_index].argmax()))
For classification, tagging or retrieval, fine-tune the encoder on your task as you would any BERT-style model.
Fit
Where NandiBERT Fits
Reach for NandiBERT-150M when latency and memory matter: high-volume classification, tagging and reranking, on hardware you own. Reach for NandiBERT-450M when inputs are long or Indic accuracy matters most. Both are a strong place to start a fine-tune in any of the languages they cover, because one model serves all of them.
Know their edges. These are base encoders, so they need fine-tuning for most tasks, and they do not generate text. They are not sentence-embedding models yet; that is what EmbeddingNandi is for. And in English alone, an English-only specialist still leads the average in each size class.
The Takeaway
NandiBERT is one encoder for English, Indic, Bahasa Indonesia, Malay and Arabic. It is the state of the art in Indic, among the best in English against English-only specialists, it reads whole documents, and its weights are open for you to run on your own infrastructure. That is where Sovereign Superintelligence starts.