NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Summary

The paper introduces NeoMME, a family of 260M and 800M-parameter bidirectional Transformer encoders trained entirely from scratch to process multilingual text tokens and raw 32x32 image patches in a single tower, without a pretrained vision encoder or causal decoder. Pretraining uses an image-conditioned masked discrete-diffusion text objective over roughly 524B packed tokens, with dynamic-resolution images, 16,384-token context, 2D RoPE, and interleaved sliding-window/global attention. Fine-tuned with jointly trained dense and late-interaction heads, NeoMME-Retriever reaches 0.523 (260M) and 0.556 (800M) nDCG@10 on ViDoRe v3; the 260M model outperforms all evaluated models strictly below 800M parameters, and at matched 2048x2048 input it encodes pages about twice as fast as ColModernVBERT on an L40S. Hierarchical token pooling plus int8-query/binary-document quantization compresses the late-interaction index 255x while retaining 95.19% of nDCG@10. Transfer is solid on document and text tasks (75.3 mean on a 17-task suite) but weak on frozen natural-image probes (13.2 average accuracy), which the authors attribute to the absence of any image-prediction objective. Strengths include Apache-2.0 checkpoints with predecay releases, day-zero Hugging Face Transformers integration, and unusually candid limitation reporting: roughly seven times less text pretraining data than ModernBERT, tokenizer compression about 65% worse than mmBERT across all 204 FLORES-200 languages, and retrieval supervision far below the strongest text baseline. Key weaknesses: no ablations isolating architectural choices, no significance testing, and a dense head that lags its own late-interaction head by roughly 13 nDCG points.