MODULE 03 — BEYOND TEXT · 11 AI Concepts

Multimodal AI

A photo of a dog, the word “dog”, and the sound of barking arrive through three different doors — and land in the same neighbourhood.

Send one thing in, four ways

image audio text video

Each door has its own encoder — for an image, that means scanning the pixels and gradually boiling them down to "what's in the picture," the same way page 2 boiled a sentence down to chunks. What comes out the other side is the same kind of thing either way: a point in the shared space.

The problem it solves

Formats used to be islands

BEFORE — ONE STORE PER FORMAT photos text audio no way to ask “which photo matches this sentence?” NOW — ONE SHARED SPACE everything “dog” everything “car” distance now works between any two things, whatever their format

How it works

Different doors, identical output

image audio text encoder encoder encoder built for one format each
Its own doorPixels, waveforms and letters need genuinely different handling.
encoder encoder encoder [0.21, −0.07, …] same shape, always
Identical outputEvery door produces a list of numbers of the same length.
near far a photo and a sentence can now be compared
Distance works across formats“Find photos of a dog on a beach” becomes a nearest-point search.

The modern version

Or: skip the doors entirely

CUT EVERYTHING INTO TOKENS AND INTERLEAVE a photo a sentence cut into patches cut into chunks one stream · picture and words side by side one model the next-word machine again no separate encoders, no shared map — just more kinds of token

The shared-space picture above is the intuition. Today’s big models mostly do this instead — which is why a picture and a question about it fit on the same desk.

So what

Searching your photos by typing a sentence is not clever image recognition. It is one nearest-point lookup, in a space that stopped caring what format things arrived in.