Multimodal AI: Vision, Audio & Images
Features that see, hear, speak, and draw
Add images, documents, audio, and video to your product: vision input, document understanding, image generation and editing, speech-to-text, text-to-speech, realtime voice, video, and multimodal search — plus the media pipelines behind them.
- Lessons
- 9
- Reading time
- 90 min
- Level
- Intermediate
- Access
- Full course
Lessons in this module
- 7.112 min
Adding Vision to Your App
Send images to a model and get useful answers back: input formats, resolution and cost, prompting for visual tasks, and the things vision models still miss.
- 7.211 min
Document Understanding: Receipts, Forms, and PDFs
Read invoices, receipts, IDs, and multi-page PDFs into structured data — native PDF input vs. OCR, page-level citations, and human review for edge cases.
- 7.311 min
Image Generation and Editing in Products
Add text-to-image and image editing to an app: choosing a model, prompt templates for consistent results, storage, moderation, and licensing questions.
- 7.49 min
Speech-to-Text Features
Transcription for meetings, voice notes, and dictation: model options, diarization, timestamps, long files, and post-processing with an LLM.
- 7.59 min
Text-to-Speech and Voice Output
Give your product a voice: TTS models and voices, streaming audio, pronunciation control, and the consent rules around voice cloning.
- 7.610 min
Realtime Voice Interfaces
Speech-to-speech models vs. chained STT-LLM-TTS pipelines, latency budgets, turn-taking and interruptions, and when voice is the right interface at all.
- 7.79 min
Video Understanding and Generation
Analyze video by sampling frames and transcripts, summarize and search long recordings, and where generative video fits in a product today.
- 7.89 min
Multimodal Search with Image and Text Embeddings
Search photos by description and find similar images with multimodal embeddings — plus captioning as a simple, powerful alternative.
- 7.910 min
Media Pipelines: Uploads, Storage, and Cost
The plumbing behind multimodal features: direct uploads, validation, resizing, object storage, async processing, and keeping media costs predictable.
About this module
Multimodal AI means models that work with more than text: they read images, screenshots, and PDFs, transcribe and speak audio, generate and edit pictures, and analyze video. For product builders, that turns the attachments, voice notes, and photos your users already send into input your features can understand, and lets your product answer with images and speech instead of only paragraphs.
It matters because so much real information isn't text. The error code is in a screenshot, the total is on a photographed receipt, the decision was made in a recorded meeting. A text-only AI feature can't see any of it. Multimodal features also bring new costs and failure modes: images are billed by resolution, audio by the minute, and a misread digit or a garbled name looks just as confident as a correct one.
This module covers the full range of multimodal features you can ship today. You'll add vision input to an app and learn how resolution drives cost, read invoices, receipts, and multi-page PDFs into structured data with page-level evidence, add image generation and editing with consistent prompt templates and moderation, build transcription with diarization and timestamps, give your product a voice with text-to-speech, and decide when a realtime voice interface is worth its latency budget.
It finishes with video understanding through frame sampling and transcripts, multimodal search with image and text embeddings, and the media pipelines behind all of it: direct uploads, validation, resizing, storage, async processing, and keeping media costs predictable. Every lesson uses real code, current models, and honest guidance on where multimodal AI still falls short.
Questions about Multimodal AI: Vision, Audio & Images
What is multimodal AI?
Multimodal AI refers to models that accept or produce more than one kind of data, such as text, images, audio, and video. Current flagship language models can take images and PDFs as input alongside text, while separate specialized models handle speech recognition, text-to-speech, and image generation.
How much does it cost to send an image to an LLM?
Images are billed as input tokens based on their pixel dimensions, not their file size. On Anthropic's API, an image costs roughly one token per 28 by 28 pixel patch, so a full-HD screenshot is a few thousand tokens. Resizing or cropping images before sending them is the main way to control the cost.
Should I use OCR or an LLM to extract data from PDFs?
Use plain text extraction for born-digital PDFs with a clean text layer, because it is cheap and fast. Use a vision-capable model with native PDF input for scans, photos, complex tables, and varied layouts. Dedicated OCR and document-AI services remain a strong choice for high-volume, fixed-template forms.
Can Claude generate images or transcribe audio?
Claude can read images and PDFs, but Anthropic does not offer image generation, speech-to-text, or text-to-speech models. Products that need those capabilities combine Claude with models from other providers, which this module covers.
When is voice the right interface for an AI product?
Voice works best when users' hands or eyes are busy, when speaking is faster than typing, or for accessibility. It is a poor fit for dense information, noisy environments, or tasks where users need to review and edit output. Realtime voice also needs a tight latency budget to feel natural.