Hugging Face Transformers v5.19.0 adds EmbeddingGemma 2 multimodal embeddings & expert-parallel dispatch
libraryTransformers 5.19.0 ships EmbeddingGemma 2, a Gemma-4-based multimodal embedding model that encodes text, images, audio, and video into a shared 768-d space with Matryoshka truncation to 512/256/128 dims and configurable vision/audio towers. The release also adds expert-parallel token dispatch (default for Qwen3 MoE/Mellum), per-layer cache configuration, continuous-batching support in regular SDPA/flash attention, and deprecates the "paged|" attention prefix.
An agent can now produce unified multimodal embeddings across text, images, audio, and video with EmbeddingGemma 2 via Transformers, with truncatable Matryoshka dims for cheaper cross-modal retrieval and clustering.
Major release: new multimodal embedding model + MoE expert-parallelism improvements. Note breaking changes to MoE router_logits outputs, OWLv2 image-query selection, and deprecation of "paged|" attention prefix for CB.
pip install transformers==5.19.0from transformers import AutoModel, AutoProcessor
import torch
model_id = "google/embeddinggemma-2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
# Encode a mixed text+image input
inputs = processor(
text=["a photo of a cat on a sofa"],
images=["cat.jpg"],
return_tensors="pt",
).to(model.device)
with torch.no_grad():
embeds = model(**inputs).embeddings # (1, 768)
# Matryoshka truncation to 256 dims
embeds_256 = torch.nn.functional.normalize(embeds[:, :256], dim=-1)
print(embeds_256.shape)