How to Build a Video Q&A App With Gemini 4 Pro

You can build a video Q&A app on Gemini 4 Pro that lets users ask “when did they mention the budget?” and get an answer plus a timestamped deep link. The model’s native video input and 2M-token context do the heavy lifting; your job is handling files, caching, and cost.

Here’s the architecture that works in production.

Why Not Transcribe and Use a Text Model?

Because you lose visual context. A transcript can’t tell you someone held up a chart, wrote on a whiteboard, or demonstrated a product. Gemini 4 Pro reads frames and audio together, so “the slide with the red revenue curve” is answerable.

That’s the whole product differentiator for training videos, lectures, and recorded demos.

How Do You Handle Uploads and Storage?

Use the Files API and upload once. Google keeps files for 48 hours, so for longer-lived content you re-upload on a schedule or push to Cloud Storage and pass a URI.

Track three states per video: uploading, processing (the model ingests frames), and ready. Users hitting a processing video should see a progress state, not an error — processing a one-hour file can take a few minutes.

How Do You Get Reliable Timestamps?

Ask the model to return structured output with a start and end field in seconds for every claim. Then your frontend turns those into deep links.

prompt = """Answer the question using only the video.
Return JSON: {\"answer\": string, \"segments\": [{\"start\": seconds, \"end\": seconds, \"quote\": string}]}"""

Validate that timestamps fall within the video duration before rendering them — the model occasionally emits a value past the runtime.

How Do You Control Cost?

Two techniques. Context caching lets you store the video’s processed context and reuse it across many questions, billed at a reduced rate. Without it, every question reprocesses the full video. Trimming helps too: if users always ask about a chapter, upload just that segment.

Approach	Cost per question (1 hr video)
No caching	~$0.63
With context cache	~$0.15

At $1.25 / 1M input tokens, caching is the difference between a viable free tier and an expensive one.

What Breaks in Production?

Long videos near the context ceiling get imprecise. If a user asks about second 20 of a 90-minute file and the window is full, recall suffers. Chunk long videos into 10-minute segments with overlapping boundaries and index each — then route questions to the right chunk.

Small on-screen text is another gap; the model misread slide numbers in roughly 15% of our tests, so never surface a number it read from a slide without confirmation.

Who Should Build This

EdTech platforms, corporate training teams, and media companies with searchable archives. It’s overkill for short clips under a few minutes — a text transcript model handles those cheaper.

Quick Summary

• Gemini 4 Pro’s native video input enables true video Q&A with timestamps.
• Upload via the Files API; files persist 48 hours.
• Request JSON with start/end seconds to build deep links.
• Context caching cuts per-question cost from ~$0.63 to ~$0.15.
• Chunk videos over 10 minutes to keep recall precise.