Capstone 3: A Multimodal Expense App

Expense reports are a perfect multimodal problem: the input is a crumpled receipt photographed at an angle, a PDF invoice from a hotel, or a voice note recorded in a taxi ("lunch with the Acme folks, about sixty bucks, yesterday"). The output has to be exact, because someone gets reimbursed from it. A model that reads 95% of receipts correctly is impressive; an app that silently files the other 5% with the wrong total is a liability.

In this capstone you'll build an expense app where users snap a receipt, upload a PDF, or speak a note and get a categorized draft expense back. You'll combine vision extraction, speech-to-text, validation rules in code, a review screen that puts the user's attention where it's needed, and the media pipeline underneath it all.

What you'll build#

A small web app (Next.js and Postgres) with one core flow:

1. The user uploads a receipt photo, a PDF, or a voice note from their phone.
2. The server identifies the file by its bytes, normalizes images, stores the media privately, and responds immediately.
3. In the background, a model extracts a draft expense: merchant, date, currency, total, tax, tip, line items, and category. Voice notes are transcribed first.
4. Code validates the draft: missing fields, impossible dates, totals that don't add up, policy limits, likely duplicates.
5. The user sees the source next to the extracted fields, with anything doubtful highlighted, and approves.

 phone ──POST /api/expenses (multipart)──▶ sniff bytes → size limit
                                            │ photo: auto-orient, resize, JPEG, strip EXIF
                                            ▼
                                   storage (private)   expenses row: processing
                                            │
                       after() / queue      ▼
                ┌───────────── processExpense() ─────────────┐
                │ photo/PDF ──▶ Sonnet 5.5 vision ─┐         │
                │ voice ──▶ transcription ──▶ Haiku 5.5 ─┤   │
                │                                  ▼         │
                │            ExpenseDraft (structured output)│
                │                                  ▼         │
                │  validateDraft() + duplicate check (code)  │
                └──────────────────────┬─────────────────────┘
                                       ▼
                        status: ready | needs_review | failed
                                       ▼
                     review screen: source + fields + flags → approve

The design principle running through every step: the model proposes, code checks, the user decides. The model is good at reading messy inputs. It is not the place to enforce policy, do arithmetic you'll pay out on, or decide what counts as a duplicate.

This build leans on Adding Vision to Your App, Document Understanding: Receipts, Forms, and PDFs, Speech-to-Text Features, Media Pipelines, and Validation, Retries, and Repair.