Research Briefings - Gemma 4

Released April 2, 2026, Gemma 4 is Google DeepMind’s most capable open model family, built on the same research as Gemini 3 and released under a fully permissive Apache 2.0 license.

It comes in four sizes:

• E2B and E4B are edge-optimized models that activate an effective 2B and 4B parameters, designed to run offline on phones, Raspberry Pi, and Jetson Nano.

• 26B-A4B (MoE) is a mixture-of-experts model with 26B total parameters but only 3.8B active per token, so you get 26B quality at 4B speed.

• 31B Dense is the largest, currently ranked #3 globally among open models on Arena AI.

Key capabilities across the family:

• native vision and audio processing, function calling,

• structured JSON output, system instructions,

• a thinking/reasoning mode,

• 256K context windows for the larger models (128K for edge),

• training on 140+ languages.

The benchmark leap is dramatic. Compared to Gemma 3, the 31B Dense model jumped from 20.8% to 89.2% on AIME math, nearly tripled LiveCodeBench scores, and went from 19% to 74% on BigBench Extra Hard. This makes it far more competitive with Qwen 3.5, which previously dominated math and coding among open models at this size class.

Architectural innovations include a second embedding (Per-Layer Embeddings) table that feeds residual signals into every decoder layer, and shared key-value tensors in the final layers to reduce memory during long-context inference.

To test the capabilities of the Gemma-4 model, I did some quick experiments. I am primarily interested in multi-modal and visual-language capabilities of the models. Hugging Face team did various tests on Gemma-4 model and wrote in a blog post. I did some further exploration of this series of models which are discussed below.

The image reasoning of Gemma-4 is tested across three experiments testing capabilities the HuggingFace blog never benchmarked. These are multi-image reasoning, adversarial robustness, and structured extraction.

Three prompts, each with two images. The question: does the model actually attend to both images, or does it default to one and confabulate the other?

• Chart Comparison (H1 vs H2 2025)

  

  Result: Correctly identified May ($61M) as the H1 peak and December ($102M) as the H2 peak. Correctly judged H2 as the better-performing half and noted that “every single month in H2 (except August) outperformed the best month of H1.” Accurately characterized H1 as volatile and H2 as trending upward.

  Verdict: ✅ Both images attended to. Numbers extracted correctly from both. Cross-chart comparison was substantive, not generic.

• Product Spec Comparison

  

  Result: Produced a structured markdown comparison table covering price, price-per-GB-RAM, storage, display, battery, and weight — declaring a winner per row. Notably, it computed $54.12/GB (UltraBook) vs $45.28/GB (ThinkPad) — actual division, not vibes.

  Verdict: ✅ Strong. It performed real arithmetic across two images and structured the answer in a table without being asked. The price-per-GB calculation is the kind of derived metric that proves the model isn’t just OCR’ing — it’s reasoning across the inputs.

• Scene Comparison (Venice + Bangkok)

  

  

  Result: Identified Venice from the seagull-on-mooring-pole and the “Ferrovia” sign (Santa Lucia train station). Identified Bangkok from the temple architecture (Grand Palace / Wat Phra Kaew). Both identifications used specific visual cues, not generic guesses.

  Verdict: ✅ The “Ferrovia” callout is impressive, it noticed text in the background and used it as evidence. This is grounded reasoning, not pattern-matching.

Summary: The model doesn’t ignore secondary images, doesn’t confuse them, and can perform calculations and comparisons that span both. This is a meaningful capability the HF blog never demonstrated.

Five prompts designed to fool the model. The question: does it confidently get things wrong, or does it flag the problem?

• Rotated Text (180°)

  Setup: A receipt image rotated upside-down.

  

  Result: Returned valid JSON with both items, both prices, and the correct total $145.26. Did not mention or notice the rotation.

  Verdict: ✅ Read the rotated text perfectly. Either the vision encoder is rotation-invariant or it normalized the image internally. Either way, the user gets the correct answer with zero friction.

• Misleading Chart (Truncated Y-axis)

  Setup: Bar chart with values 97, 98, 99, 100 but Y-axis starting at 95, visually exaggerating tiny differences.

  

  Result: Read all four values correctly. Computed the actual difference as 3 points. And then unprompted, called out: “Yes. The chart uses a truncated y-axis (it starts at 95 instead of 0)...”

  Verdict: ✅✅ This is the standout result. The model didn’t just answer the literal question — it flagged the visual deception, which is exactly what a human analyst would do. This is data literacy, not just OCR.

• Low-Resolution Degraded Image

  Setup: The bird photo downsampled to 48×48 then upscaled — pixelated and barely recognizable.

  

  Result: Opened with: “The image provided is extremely low-resolution and heavily pixelated, making it very difficult to identify specific details with certainty.” Then made a hedged guess: “likely a bird, perched on top of a grey post or pillar.”

  Verdict: ✅ Calibrated uncertainty. Many vision models confidently hallucinate detail on degraded inputs. Gemma 4 explicitly acknowledged the limitation before guessing and the guess was correct.

• Stroop Test (Color vs Text)

  Setup: Red square with the word “BLUE” written on it.

  

  Result: “The background of the image is red. The text written on it is ‘BLUE’. No, the text and the color are not consistent.”

  Verdict: ✅ Three-for-three: identified the actual pixel color, read the contradicting text, and noted the inconsistency. No confusion between symbol and substance.

• on-Latin OCR (Japanese)

  Setup: An image with two lines of Japanese text:

  • 東京タワーの高さは333mです (Tokyo Tower’s height is 333m)

  • 建設年: 1958年12月23日 (Construction year: December 23, 1958)

  

  Result:

  Transcription: 333m 1958 12 23

  Translation: 333m December 23, 1958

  Verdict: ❌ The only clear failure. The model extracted only the numbers (333m, 1958/12/23) and dropped all the Japanese characters. It got the dates and measurements right, but lost the semantic context — it never tells you this is about Tokyo Tower or that 1958/12/23 is a “construction year.” The Japanese script was effectively ignored.

Six prompts covering captioning, object detection, OCR, spatial reasoning, and chart extraction.

• Detailed Captioning (Bird Photo)

  

  Result Produced a single rich sentence: “A single seagull with grey and white plumage stands perched atop a weathered grey wooden mooring pole in the foreground of a bright, daylight scene...” — included subject, environment, lighting, and architectural background details.

  Verdict: ✅ Reads like professional photo caption copy. No hallucinated objects.

• Object Detection Bounding Box (Bike)

  

  Result: [{"box_2d": [542, 0, 954, 893], "label": "the bike"}]

  Verdict: ✅ Valid JSON, single object, coordinates in the documented 1000×1000 normalized space. Coordinates need visual verification but format is correct.

• GUI Element Detection (View Recipe Button)

  

  Result: [{"box_2d": [171, 104, 246, 308], "label": "view recipe"}]

  Verdict: ✅ Compare to the HF blog’s reference for the same image: [171, 75, 245, 308]. Within ~30 pixels, which basically matches. This demonstrates reproducibility against Google’s reference outputs.

• OCR + Structured Extraction (Receipt)

  

  Result:

  Verdict: ✅✅ Perfect extraction. Numeric values are typed as numbers (not strings), spacing in product names was correctly inferred (Widget Pro not WidgetPro — even though the rotated-text version in Experiment 2 had WidgetPro without the space; the upright version got it right). All four numeric fields correct.

• Spatial Reasoning

  

  Prompt: What is to the left of the bird? Behind it? Below it?

  Result: Correctly placed “white building with windows and trees” to the left, “clear, light-colored sky” behind, and “weathered grey wooden piling” below.

  Verdict: ✅ Concise, anatomically structured answer. Real spatial grounding rather than generic scene description.

• Chart Data Extraction

  

  Result:

  Verdict: ✅ Six values, all correct, valid JSON, ready to use as a data source.

To explore the video understanding capabilities, I used a personal video which is posted as a reel on my Instagram. This is a picnic-and-reading-in-a-park video, trimmed to 60s.

• Basic Description

  The 31B gave a remarkably accurate scene-by-scene description:

  “A man and a woman are enjoying a sunny day outdoors in a grassy park area with trees and a distant body of water... The video begins with a stylized sequence of the two lying on their backs in the grass, holding open books above their faces... The man is in a grey t-shirt, and the woman is in a blue patterned shirt... The camera cuts to a close-up of a hand holding a black pour-over coffee dripper over a glass, filtering coffee.”

  Verdict: ✅ Highly accurate. It picked up the opening symmetry shot, the wardrobe colors, the pour-over coffee detail, and the narrative arc (reading → conversation). No hallucinations.

• Structured Categorization

  Returned a clean, parseable JSON object:

  Verdict: ✅✅ Production-ready. Every field is filled correctly, the audience targeting is specific (not "everyone"), and the JSON parsed on the first try.

• Temporal Understanding

  The model produced a precise 5-segment timeline. It also correctly identified the editing technique (hard cuts, jump cuts between 00:03-00:15, classic B-roll cutaway to coffee at 00:35) and called the pacing “slow and rhythmic, matching the relaxed vibe of a picnic.”

  Verdict: ✅✅ This is the most impressive output of the whole experiment. The model didn’t just describe frames — it understood video grammar. Cutaways, jump cuts, narrative arc, pacing changes. For an Instagram pipeline, this temporal segmentation is exactly what you need to detect things like “the hook is too long” or “the CTA is missing.”

• Video + Audio Description

  The E4B model (8B params with embeddings) is the only Gemma 4 variant that processes audio from video. We run the same videos and compare what the audio track adds.

  The good: It correctly identified that there’s spoken dialogue (no music), correctly inferred the speakers were “talking about their decision to dedicate a time to reading after falling behind,” and noted “a beautiful spring, sunny day.”

  The bad: It transcribed the location as “Hamsterhiif” which is a hallucination of a real place name Hampstead Heath. (Across the three experiments it variously called it “Hamsterhiif”, “Hamsteriheb”, and “Hamsteriheb” — the model couldn’t even agree with itself across runs.)

  The very bad: It hallucinated visual content that isn’t in the video. From its description:

  “They are observed helping to set up or work with a large, tent-like structure, possibly related to a field activity or community project. They appear to be engaged in some sort of manual labor.”

  This appears to be a misinterpretation of the blanket-spreading sequence the 31B correctly identified in segment 00:16–00:23. The E4B turned “spreading a picnic blanket” into “manual labor on a tent structure.” The 2b ablation reinforces this: the audio-OFF run later described “covering a large, canvas-like structure with a printed design of stylized trees” — pure confabulation.

  Verdict: ⚠️ Audio detection works (speech vs music classification is correct). But specific transcription is unreliable, and the visual descriptions are noticeably worse than the 31B’s. The location name fabrication is the kind of error that would silently corrupt downstream analytics.

This look quite promising. I have a whole range of experiments that I am running on image and video understanding capabilities of these models. And I’ll share more details on a project that I am working on related to video reasoning of large (visual) language models.

Codes used for this experiment can be found at GitHub: januverma/gemma4-experiments