Gemma Explained: Key Architecture Changes and Features in Gemma 3

ArticleJu-yeong Ji, Ravin KumarResource

Google outlines the architectural changes in Gemma 3, highlighting vision-language capabilities, longer context windows, an expanded tokenizer, and memory-saving attention mechanisms.

Key points

  1. Vision-language capabilities: The 4B, 12B, and 27B models include a custom SigLIP vision encoder operating at 896x896 resolution, using a Pan&Scan approach for arbitrary aspect ratios.
  2. Soft image tokens: A MultiModalProjector converts image inputs into 256 compact soft token vectors, applying bidirectional attention to visual data and unidirectional attention to text.
  3. 5-to-1 interleaved attention: Repeating blocks of five local sliding-window layers (window size 1024) and one global layer reduce KV-cache memory consumption.
  4. Transition from soft-capping to QK-norm: Gemma 3 drops Gemma 2's soft-capping mechanism in favour of QK-norm for faster throughput and improved accuracy.
  5. Extended context: Context length reaches 32k tokens on the 1B model and 128k tokens on larger models.
  6. Updated tokenizer: An expanded 262k vocabulary matching Gemini provides improved balance for non-English languages.

Why it matters

Gemma 3 brings native multimodal support and 128k context windows to open weights models while mitigating memory overhead. The shift to interleaved attention and QK-norm makes larger context execution more viable on constrained hardware.

Analysis

Authors Ju-yeong Ji and Ravin Kumar clarify the distinction between Gemma 3 and PaliGemma 2: PaliGemma 2 remains focused on detection and segmentation tasks, whereas Gemma 3 prioritises multi-turn chat, zero-shot visual reasoning, and long-context processing. Treating visual inputs with bidirectional attention while compressing them into 256 tokens strikes a balance between visual fidelity and inference compute requirements.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:Z.ai

    Z.ai Releases GLM-5.3-Flash as an Open-Weights Multimodal Model

    Z.ai has launched GLM-5.3-Flash under the MIT licence on Hugging Face, marking the first natively multimodal model in the GLM-5 family. Featuring 320B total parameters and 18B active parameters, it targets high efficiency across coding and agentic tasks.

    AI & LLMs· Tool

  • Article:Kimi Team (Yu Zhang et al.)

    Kimi Linear: An Expressive Hybrid Linear Attention Architecture

    The Kimi Team presents Kimi Linear, a hybrid linear attention architecture that outperforms standard full attention across short-context, long-context, and RL scaling scenarios. It cuts KV cache consumption by up to 75% and boosts decoding throughput up to sixfold on 1M token contexts.

    AI & LLMs· Research

  • Link:Moonshot AI

    Moonshot AI Releases Kimi K3: Open-Weight 2.8T MoE Model

    Moonshot AI has released Kimi K3, an open-weight native multimodal agentic model featuring 2.8 trillion total parameters and a 1-million-token context window. The architecture uses a sparse MoE setup that activates 104 billion parameters per token.

    AI & LLMs· Announcement

  • X post:Mia

    Google Releases EmbeddingGemma 2: Unified Multimodal Vector Space

    Google has launched EmbeddingGemma 2, an open-weights model mapping text, code, images, video, and audio into a single shared vector space. The 1.49 GB model is released under the Apache 2.0 licence.

    684bookmarks37kviews

    AI & LLMs· Announcement

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.