Video Understanding and Generation
Analyze video by sampling frames and transcripts, summarize and search long recordings, and where generative video fits in a product today.
Video is where a lot of your users' most useful information is trapped. Customers record their screens to show a bug instead of describing it. Teams record every meeting and rarely rewatch any of them. Training libraries grow to hundreds of hours nobody can search. Video is also the most expensive and awkward medium to feed a model: a ten-minute recording at 30 frames per second is 18,000 images plus an audio track.
The good news is that you almost never need all of it. In this lesson you'll learn the three ways to get video into a model (transcript, sampled frames, native video input), build a Deskmate feature that turns a customer's screen recording into reproduction steps, summarize and search long recordings, and get a realistic picture of where generated video fits in a product today.
What video features actually need
Before choosing a technique, ask what information in the video matters. Most video features rely mainly on one of two tracks:
| Feature | Where the information is | What you need from the video |
|---|---|---|
| Meeting and call summaries | Almost entirely the audio | A good transcript with speaker labels |
| Training and tutorial search | Audio, plus slides or demos | Transcript, plus occasional frames |
| Bug reports from screen recordings | Mostly the visuals | Frames at the right moments, plus any narration |
| Moderating user uploads | Visuals and audio | Sampled frames across the whole video, plus a transcript |
| Sports, security, or process footage | Motion over time | Dense frames, sometimes specialized models |
That first column decides your costs. A meeting summary built from the transcript alone costs a fraction of one that looks at frames, and is usually just as good. Don't send pixels to a model that only needs words. You learned to produce those transcripts in Speech-to-Text Features.
Our example is Deskmate's screen-recording bug reports. A customer attaches a two-minute recording to a ticket; Deskmate turns it into timestamped reproduction steps an engineer can follow without watching the whole thing. Here the visuals are the point: the customer may not say a word.