Vision-language model

A vision-language model, often shortened to VLM, pairs an image encoder with a language model. You send one or more images plus a text instruction, and it replies in text: a description, an answer, extracted fields, or a judgment. Current Claude, GPT, and Gemini models all accept images, and there are open-weight VLMs you can host yourself.

In products, VLMs appear wherever users already have pictures: bug screenshots attached to support tickets, photos of damaged deliveries, receipts, whiteboards, product images that need tags or alt text. Combined with structured outputs, a VLM turns a picture into data your app can act on. It also understands layout, which is why it often beats plain OCR on documents where position carries meaning.

Images are billed as tokens based on their size, so downscale to the smallest resolution that still answers the question. Fix orientation and format before sending, and put the image before your question in the prompt. VLMs remain weaker at counting, precise spatial reasoning, fine print, and reading exact values off charts, and they can confidently describe things that aren't in the image. Test on messy real uploads, and keep a person in the loop when a misread would be costly.