10. Multimodal AI: Images, Audio, Video, and Files

Design multimodal inputs, handle file safety, process audio and images, and build accessibility-aware experiences.

Image Inputs and Vision Workflows

Multimodal models may accept text and images in one request, but support and image handling differ by model and provider. Images can be resized or tiled internally, and high-resolution analysis may cost more. Use images only when they add task-relevant evidence; avoid sending unrelated or sensitive images.

For image workflows, validate content type and size, remove unnecessary metadata where appropriate, check user consent and access, and store files securely. The model may miss small text, exact counts, or subtle visual details. For exact measurements or identity-sensitive tasks, use dedicated deterministic or human-reviewed workflows as appropriate.

Diagram

Image workflow

Upload -> authenticate -> validate type/size -> secure storage
       -> create provider-compatible input
       -> model analysis -> validate response
       -> display answer with uncertainty

💡 Do not trust a file extension alone to establish file type. Use server-side validation and limit the model to the image data required for the task.

Key Takeaways

Validate uploads and minimize sensitive image exposure.
Choose model/image settings based on task needs.
Use human review for high-impact visual judgments.

Knowledge Check

What is an important limitation of image understanding?

Speech-to-Text, Text-to-Speech, and Realtime Audio

Audio applications may transcribe speech, synthesize speech, classify sounds, or support low-latency conversational voice. Decide whether the product needs batch transcription, streaming partial transcripts, turn detection, interruption handling, or bidirectional audio. Audio codecs, sample rates, chunk boundaries, and session protocols must match the provider's requirements.

Build for noisy input, accents, language switching, and transcription uncertainty. For voice actions, do not treat transcription as proof of intent for a high-impact operation; confirm important details and enforce the same authorization as text interactions. Provide accessible text alternatives and clear recording indicators.

Audio application stages

Microphone -> permission + capture -> audio transport
          -> transcription / realtime model
          -> intent + policy check
          -> response text / synthesized speech
          -> playback + interruption handling

💡 The correct architecture depends on latency requirements. A batch pipeline is simpler; realtime voice needs session lifecycle, cancellation, partial results, and interruption management.

Key Takeaways

Match audio transport to latency requirements.
Design for transcription uncertainty and interruptions.
Use consistent authorization across modalities.

Knowledge Check

Why should a high-impact voice action require the same authorization checks as a text action?

PDFs, Office Files, and Document Understanding

Document AI often combines file parsing, OCR, layout extraction, chunking, and model reasoning. Preserve page numbers, section headings, tables, and source offsets where possible so answers can cite evidence. OCR and layout extraction can introduce errors, particularly with scans, columns, handwriting, or complex tables.

Apply upload limits, malware scanning where appropriate, content-type validation, access controls, and retention rules. Treat file contents as untrusted instructions and do not let embedded document text override application policy. For large files, use asynchronous ingestion with progress and retry states.

Document ingestion pipeline

File -> validation / malware scan -> parser / OCR
     -> normalized text + page metadata
     -> chunks + embeddings -> retrieval index
     -> question-time retrieval -> cited answer

💡 This pipeline separates expensive file processing from interactive question answering. Keep parser version and page provenance to debug incorrect extraction or citations.

Key Takeaways

Preserve layout and provenance when citations matter.
Treat document contents as untrusted data.
Use asynchronous processing for expensive large-file ingestion.

Knowledge Check

Why preserve page or section provenance during document ingestion?

Image Inputs and Vision Workflows

Speech-to-Text, Text-to-Speech, and Realtime Audio

PDFs, Office Files, and Document Understanding

STAY UPDATED WITH BYTES!