Adding Vision to Your App
Send images to a model and get useful answers back: input formats, resolution and cost, prompting for visual tasks, and the things vision models still miss.
"See attached screenshot." Every support team knows that line. The ticket body says almost nothing, and the useful information (the error code, the page the customer was on, the button that's greyed out) is in a PNG that nobody reads until a human opens the ticket. Your text-only AI features can't see it. Your triage classifier routes the ticket blind, and your ticket summary has a hole in the middle.
Vision models close that gap. You send an image alongside text and get back an answer, a transcription, or a structured record. In this lesson you'll add screenshot understanding to Deskmate's tickets, and along the way learn the decisions that matter for any vision feature: how to send images, what resolution costs, how to prompt for visual tasks, and the specific ways vision models still get things wrong.
Where vision earns its place in a product
Most product uses of vision fall into a handful of patterns. Naming the pattern first tells you what the output should look like and how to check it.
| Pattern | Example | Output | How you check it |
|---|---|---|---|
| Transcribe | Copy the error message out of a screenshot | Exact text | Compare to the image; text should be verbatim |
| Describe | Alt text for an uploaded image; a caption for search | Short prose | Human spot checks |
| Classify | Is this a screenshot, a photo of a screen, or a document? | A label from a fixed set | Labeled examples, accuracy |
| Answer about | "Which plan is selected on this billing page?" | A short answer, often structured | Labeled examples |
| Compare | Does the product photo match the listing? | A verdict plus reasons | Labeled pairs |
A vision model here just means a language model that accepts images as input. All current Claude models do, and so do the flagship models from OpenAI and Google. You don't need a separate "vision API."
Before you reach for one, ask whether the image is the cheapest source of the information. If your app can capture the page URL and error code automatically when a user files a ticket, that metadata is exact and free. Vision is for information that only exists as pixels, in a format you don't control: Deskmate customers paste screenshots from your app, their email client, even phone photos of a monitor.