L5 · 04 · Multimodal AI: Vision, Speech & Video
So far every model in this curriculum has been text-in, text-out. Multimodal models see, hear and generate more than words — images, audio, video — and are what make AI useful in the physical world. This page maps the modalities, how they're fused into one model, and the practical uses and limits of each.
By the end of this page you can
- Explain how images/audio become tokens and share a model with text.
- Describe vision-language models (VLMs) and their uses (document understanding, UI, OCR).
- Describe speech: ASR, TTS and real-time voice; and video understanding.
- Describe generation modalities: diffusion for images, and beyond.
- Reason about the costs and failure modes specific to each modality.
One model, many senses
The trick is the same as text: turn the modality into tokens/embeddings, feed them into a shared transformer.
- Images → split into patches, a vision encoder turns each into an embedding.
- Audio → spectrogram chunks → audio encoder → embeddings.
- Text → subword tokens as usual.
All land in the same transformer, so the model can reason across modalities: "what's wrong in this screenshot and the log line under it?"
Vision-language models (VLMs)
Image + text in, text out. The most practically useful modality for IT work:
| Use case | Example |
|---|---|
| Document understanding | Extract fields from invoices, receipts, forms — layout + text together |
| UI / front-end | Screenshot → describe, generate code, find the bug visually |
| Diagrams & charts | Read an architecture diagram or a metrics graph |
| OCR+ | Not just read text, but understand it in context (tables, handwriting) |
| Accessibility | Alt-text generation, describe images for screen readers |
Speech: ASR, TTS and real-time voice
| Direction | Tech | Uses |
|---|---|---|
| Speech → text (ASR) | Whisper-style models | Transcription, meeting notes, voice input, call analytics |
| Text → speech (TTS) | Neural TTS, voice cloning | Voice assistants, accessibility, content narration |
| Real-time voice | Streaming ASR + LLM + TTS (or end-to-end audio models) | Live voice agents, hands-free ops |
Two architectures for voice agents: the cascade (ASR → LLM → TTS, easy to debug, adds latency) versus speech-to-speech models (audio in, audio out, lower latency, captures tone/emotion, newer).
Video understanding
Video = many frames + audio + time. Models sample frames and reason over the sequence. Uses: analysing a screen recording of a bug, understanding a process from a walkthrough, searching footage. The constraint is context — a long video is thousands of frames, so sampling and summarisation strategies matter.
Generation beyond text: diffusion
Text-to-image models (diffusion) work differently from LLMs: they start from noise and iteratively denoise toward an image matching the prompt. Key concepts:
- Latent diffusion: diffuse in a compressed latent space, not pixel space — that's what makes it fast enough.
- Guidance: how strongly to follow the prompt (CFG scale).
- Fine-tuning: LoRA on a style/subject (the same LoRA idea as L3·06, for images).
For IT work the value is less "art" and more: generating UI mockups, diagrams, test fixtures, and synthetic training images.
Cost & practicality
- Images are expensive tokens — a high-res image can be thousands of tokens; downscale before sending.
- Audio/video multiply context fast — transcribe or sample rather than sending raw media where possible.
- Latency: real-time voice needs streaming end-to-end; batch image analysis can be slow.
- Evals are harder: "is this image description right?" needs human or VLM-judge scoring.
- Privacy: images/voice contain biometric & personal data — extra care, often an on-device case (L5·03).
- Feed a VLM a screenshot of a UI or an architecture diagram. Where does it nail it, where does it misread detail?
- Give it a dense table image. Count the rows yourself — did it get the exact figure?
- Transcribe a voice note with a local Whisper model. Compare cost/latency vs. a hosted API.
Key takeaways
- Multimodal = encode each modality to tokens/embeddings, share one transformer.
- VLMs are the highest-value modality for IT: documents, UI, diagrams, OCR+.
- Voice agents: cascade (debuggable) vs. speech-to-speech (low-latency).
- Diffusion generates images by denoising in latent space; multimodal costs and evals are harder.