Multimodal AI — GPT-5, Gemini, and beyond

Section 10.7.3 : Multimodal AI — GPT-5, Gemini, and beyond

🎯 Learning objective

Understand what multimodal AI is — capable of processing text, images, audio, and video simultaneously — and leverage its unique professional capabilities. You'll learn which models offer which multimodal capabilities, and how to use them concretely in your workflows.

What is multimodal?

A unimodal model understands only one type of data: text → text (the first GPTs). A multimodal model can understand and generate multiple types of data in the same interaction.

AI modalities:

• →Text → text (classic LLM: Q&A, writing)
• →Image → text (vision: description, analysis, OCR, object detection)
• →Text → image (generation: DALL-E 3, Midjourney v7)
• →Audio → text (transcription: Whisper, speech recognition)
• →Text → audio (speech synthesis: TTS, ElevenLabs)
• →Video → text (video analysis: scene understanding)
• →Text → video (generation: Sora 2, Veo 2)

The multimodal revolution of 2025-2026 is that all these modalities coexist in the same model. You can send a photo, ask a voice question, and receive a text + image response in the same conversation.

Natively multimodal models in 2026: GPT-5, Gemini 3.1, Claude Opus 4.6 (vision + text), Llama 4 (vision + text + audio), and Grok 4.1 (vision + image generation via Aurora). These models process all these formats in the same conversation without requiring separate tools.

Why multimodal changes everything

Before multimodal (2023):

• →You take a photo of a chart → manually transcribe it → paste it into ChatGPT → ask for analysis
• →5 minutes of work, 3 manual steps

With multimodal (2026):

• →You upload the photo directly into GPT-5 → "Analyze this chart and identify trends"
• →10 seconds, 1 step

The impact is exponential: every eliminated manual step frees up cognitive time. When AI understands images, audio AND video, entire workflows that took hours become near-instantaneous.

Multimodal capabilities in practice

Image analysis (vision):

[Upload a photo / screenshot / chart]

Analyze this image comprehensively:
1. Factual description: describe precisely what you see
2. Data extraction: extract all numbers, text, labels (OCR)
3. If it's a chart: identify axes, trends, anomalies
4. If it's a diagram: explain the architecture and flows
5. If it's a photo: identify context, objects, atmosphere
6. Points of attention: identify problems, inconsistencies, or possible improvements

Concrete professional use cases:

GPT-5: the multimodal reference

GPT-5 (August 2025) is the most versatile model:

• →Advanced vision: real-time image analysis (via smartphone camera), complex scene understanding, handwritten document reading, chart analysis with data extraction
• →Advanced voice mode: natural voice conversation with emotions, pauses, intonation. Can adopt different tones (professional, casual, empathetic). Detects emotion in your voice and adapts
• →Native image generation: GPT-5 can create images directly in the conversation (no need to switch to DALL-E). Useful for "show me what that would look like"
• →Hybrid architecture: combines fast reasoning (for simple questions) and deep reasoning (for complex tasks), all natively multimodal

Example GPT-5 multimodal workflow:

1. →Voice: "Analyze this photo of my desk and suggest ergonomic improvements"
2. →GPT-5 analyzes the image and detects: screen too low, insufficient lighting, disorganized cables
3. →GPT-5 responds vocally + generates an image showing the reorganized desk
4. →You: "Create a shopping checklist for this reorganization with estimated prices"
5. →GPT-5 produces the list with links (if web browsing enabled)

Gemini 3.1: natively multimodal with massive window

Gemini 3.1 was designed multimodal from its architecture (not added after the fact like GPT-4):

• →2 million token window: enough to ingest an entire book (500 pages), 2h of video, or 100+ images in a single conversation
• →Native video understanding: upload a video file or share a YouTube link → Gemini analyzes both the visual and audio tracks, generates summaries with timestamps
• →Deep Think (Gemini 3 Deep Think): a deep reasoning mode specially designed for complex multimodal problems
• →Nano Banana 2: the integrated image generation module (text → image directly in Gemini)
• →Google ecosystem: native integration with Google Drive, Gmail, Calendar, Docs → multimodal + personal data

Gemini's unique use case: the 2M token window enables analyses impossible elsewhere:

[Upload a 200-page PDF — annual report]
Analyze this complete annual report:
1. Executive summary in 10 points
2. The 5 key figures to remember
3. Risks identified by management
4. Implicit comparison with the previous year
5. What is NOT said (topics avoided)

Claude Opus 4.6: vision and reasoning

Claude isn't as "fully multimodal" as GPT-5 or Gemini (no image generation or voice), but its vision is remarkable:

• →Ultra-precise image analysis: excels at OCR, complex table analysis, and understanding technical diagrams
• →200K token window: can analyze many images in a single conversation
• →Image-based reasoning: can solve math problems from exercise photos, or analyze charts with superior rigor
• →Computer Use: unique capability — Claude can "see" your screen and control mouse + keyboard to perform tasks

Advanced multimodal applications

1. Video analysis (Gemini 3.1):

[Upload a video or paste a YouTube link]
Analyze this video in depth:
- Summary in 5 key points (with timestamps)
- The speaker's main arguments, with evidence provided
- The most engaging moments (tone changes, striking statistics)
- Questions left unanswered
- If I could only watch 5 minutes, which ones?
- Follow-up suggestions: 3 complementary videos on the same topic

2. Cross-modal creation (GPT-5):

Here's a photo of my product [artisanal / tech / food / etc.].

1. Write 3 product descriptions for e-commerce (short, medium, long)
2. Identify the 3 marketing angles based on what you see visually
3. Generate a promotional image showcasing this product on a lifestyle background
4. Suggest 5 relevant Instagram hashtags
5. Write a 30-second script for a TikTok/Reel video

3. Multimodal translation:

[Upload a photo of a restaurant menu in Japanese]
1. Transcribe the Japanese text
2. Translate to English with dish descriptions
3. Identify common allergens (gluten, seafood, etc.)
4. Recommend the 3 most popular dishes for a Western palate

4. Training and education:

[Upload a photo of a handwritten math/physics exercise]
1. Transcribe the problem statement
2. Solve the exercise step by step
3. Explain each step as if I were a high school student
4. Identify the common mistakes students make on this type of exercise

Multimodal capabilities comparison table (March 2026)