Google AI Introduces Gemini Embedding 2: A Multimodal Embedding Model that Lets Your Bring Text, Images, Video, Audio, and Docs into the Embedding Space

Google expanded its Gemini model family with the release of Gemini Embedding 2. This second-generation model succeeds the text-only gemini-embedding-001 and is designed specifically to address the high-dimensional storage and cross-modal retrieval challenges faced by AI developers building production-grade Retrieval-Augmented Generation (RAG) systems. The Gemini Embedding 2 release marks a significant technical shift in how embedding models are architected, moving away from modality-specific pipelines toward a unified, natively multimodal latent space.

Native Multimodality and Interleaved Inputs

The primary architectural advancement in Gemini Embedding 2 is its ability to map five distinct media types—Text, Image, Video, Audio, and PDF—into a single, high-dimensional vector space. This eliminates the need for complex pipelines that previously required separate models for different data types, such as CLIP for images and BERT-based models for text.

The model supports interleaved inputs, allowing developers to combine different modalities in a single embedding request. This is particularly relevant for use cases where text alone does not provide sufficient context. The technical limits for these inputs are defined as:

• Text: Up to 8,192 tokens per request.
• Images: Up to 6 images (PNG, JPEG, WebP, HEIC/HEIF).
• Video: Up to 120 seconds of video (MP4, MOV, etc.).
• Audio: Up to 80 seconds of native audio (MP3, WAV, etc.) without requiring a separate transcription step.
• Documents: Up to 6 pages of PDF files.

By processing these inputs natively, Gemini Embedding 2 captures the semantic relationships between a visual frame in a video and the spoken dialogue in an audio track, projecting them as a single vector that can be compared against text queries using standard distance metrics like Cosine Similarity.

Efficiency via Matryoshka Representation Learning (MRL)

Storage and compute costs are often the primary bottlenecks in large-scale vector search. To mitigate this, Gemini Embedding 2 implements Matryoshka Representation Learning (MRL).

Standard embedding models distribute semantic information evenly across all dimensions. If a developer truncates a 3,072-dimension vector to 768 dimensions, the accuracy typically collapses because the information is lost. In contrast, Gemini Embedding 2 is trained to pack the most critical semantic information into the earliest dimensions of the vector.

The model defaults to 3,072 dimensions, but Google team has optimized three specific tiers for production use:

1. 3,072: Maximum precision for complex legal, medical, or technical datasets.
2. 1,536: A balance of performance and storage efficiency.
3. 768: Optimized for low-latency retrieval and reduced memory footprint.

Matryoshka Representation Learning (MRL) enables a ‘short-listing’ architecture. A system can perform a coarse, high-speed search across millions of items using the 768-dimension sub-vectors, then perform a precise re-ranking of the top results using the full 3,072-dimension embeddings. This reduces the computational overhead of the initial retrieval stage without sacrificing the final accuracy of the RAG pipeline.

Benchmarking: MTEB and Long-Context Retrieval

Google AI’s internal evaluation and performance on the Massive Text Embedding Benchmark (MTEB) indicate that Gemini Embedding 2 outperforms its predecessor in two specific areas: Retrieval Accuracy and Robustness to Domain Shift.

Many embedding models suffer from ‘domain drift,’ where accuracy drops when moving from generic training data (like Wikipedia) to specialized domains (like proprietary codebases). Gemini Embedding 2 utilized a multi-stage training process involving diverse datasets to ensure higher zero-shot performance across specialized tasks.

The model’s 8,192-token window is a critical specification for RAG. It allows for the embedding of larger ‘chunks’ of text, which preserves the context necessary for resolving coreferences and long-range dependencies within a document. This reduces the likelihood of ‘context fragmentation,’ a common issue where a retrieved chunk lacks the information needed for the LLM to generate a coherent answer.

Key Takeaways

1. Native Multimodality: Gemini Embedding 2 supports five distinct media types—Text, Image, Video, Audio, and PDF—within a unified vector space. This allows for interleaved inputs (e.g., an image combined with a text caption) to be processed as a single embedding without separate model pipelines.
2. Matryoshka Representation Learning (MRL): The model is architected to store the most critical semantic information in the early dimensions of a vector. While it defaults to 3,072 dimensions, it supports efficient truncation to 1,536 or 768 dimensions with minimal loss in accuracy, reducing storage costs and increasing retrieval speed.
3. Expanded Context and Performance: The model features an 8,192-token input window, allowing for larger text ‘chunks’ in RAG pipelines. It shows significant performance improvements on the Massive Text Embedding Benchmark (MTEB), specifically in retrieval accuracy and handling specialized domains like code or technical documentation.
4. Task-Specific Optimization: Developers can use task_type parameters (such as RETRIEVAL_QUERY, RETRIEVAL_DOCUMENT, or CLASSIFICATION) to provide hints to the model. This optimizes the vector’s mathematical properties for the specific operation, improving the “hit rate” in semantic search.

Check out Technical details, in Public Preview via the Gemini API and Vertex AI. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.