Multimodal Search with Image and Text Embeddings
Search photos by description and find similar images with multimodal embeddings — plus captioning as a simple, powerful alternative.
Every product that stores images eventually gets the same request: "let me search them." Users want to type "the screenshot with the red billing error" and find it, or click an image and see others like it. File names and manual tags don't get you there, because nobody names a screenshot billing-page-card-declined-error.png and nobody tags thousands of photos by hand.
There are two good ways to make images searchable, and they complement each other. Multimodal embeddings put images and text into the same vector space, so a text query can find an image directly. Captioning uses a vision model to describe each image in text, so your existing text search can find it. In this lesson you'll build both for Deskmate's ticket screenshots, learn when each one wins, and combine them.
Two ways to make images searchable
Deskmate customers attach screenshots to tickets all the time: an error dialog, a broken report, a settings page they're confused by. Support agents want two features:
- Text-to-image search: "screenshots showing a declined card on the billing page" across a workspace's tickets.
- Similar-image search: from one screenshot, find other tickets with visually similar ones. That's a fast way to spot a new bug hitting many customers at once.
Here's how the two techniques compare for these jobs:
| Multimodal embeddings | Captions + text search | |
|---|---|---|
| How it works | One model maps images and text to vectors in a shared space | A vision model writes a description; you search the text |
| Text-to-image search | Good for visual concepts ("a dark-mode dashboard") | Good, and great for exact on-screen text |
| Similar-image search | Excellent: compare image vectors directly | Weak: two captions can differ for near-identical images |
| Exact text in images | Hit or miss | Strong, if the caption copies text verbatim |
| Explainability | A distance score | Readable text you can show and debug |
| Cost per image | One embedding call (cheap) | One vision model call (more expensive) |
| Reuses existing search | Needs its own vector column and index | Plugs into the text search you already have |
Neither wins everywhere. For screenshots, where the important content is often text on screen (an error code, a page title), captions are surprisingly strong. For photos, where the content is visual (a dented package, a product color), embeddings usually win. Many production systems use both, as you'll see at the end.