Multimodal
In shortA multimodal AI model can take in, and sometimes produce, more than one kind of data, such as text, images, documents, audio or video.
In plain words
A "mode" is a kind of data. A text-only model reads and writes text. A multimodal model also handles other kinds. Anthropic, OpenAI and Google each document image input for their models, and Google's Gemini docs add video and audio. In practice you can attach a photo, a screenshot, a chart or a PDF page and ask a question about it.
Input and output are separate. Many models accept images but answer only in text. Some can also produce images or speech. Check what your model supports for both directions.
Under the hood, the model turns an image into pieces that act like tokens, so the usual rules apply: they take space in the context window and they can add to the bill. How an image is counted differs by provider, so use the provider's own counting tool for exact numbers.
Why it matters
- New tasks: read a receipt, describe a chart, extract a table from a scanned page, find a problem in a screenshot, or compare two photos.
- Cost and limits: Images and long documents use many tokens. A request with many pages can hit size limits before it hits the token limit. See price per million tokens.
- Accuracy: Models can misread small text, blurry images, dense tables or fine spatial details. Check numbers they read from images.
- Privacy: A photo or document can hold far more personal data than you notice, such as faces, addresses, metadata or other people's names. Read what never to paste into a chatbot before you upload.
- Security: Instructions can hide inside an image or file, so a model may follow text it only meant to read. OWASP lists this as a prompt injection risk that still needs more research.
Example
You photograph a printed invoice and ask: "List the items and the total as a table." The model reads the image and returns a table. You compare the total with the printed one, because a smudged digit may have been misread. You also crop out the part with the customer's address before you upload.
Common misconceptions
- "Multimodal means it sees like a human." It processes image data statistically and can miss things a person finds obvious.
- "An image costs nothing extra." It costs tokens, often many.
- "All chatbots handle all file types." Support varies by model and by app.
Questions and answers
What does multimodal mean in AI?
The model can work with more than one kind of data, such as text plus images, documents, audio or video.
Do images count as tokens?
Yes, in effect. Providers convert images into token-like units that take space in the context window and are billed. Use the provider's counter for exact numbers.
Is it safe to upload photos and scans?
Only if you are comfortable with the data leaving your device. Images can hold personal data you do not notice. Crop or remove sensitive parts first.
Read next
- Large language model (LLM)
A large language model (LLM) is an AI system trained on huge amounts of text to predict the next token, which lets it write, summarise and answer.
- Token
A token is a small chunk of text, often a word piece, that an AI model reads, writes and bills.
- Context window
The context window is the maximum amount of text, counted in tokens, that a model can read and write in one request.
- Prompt injection
Prompt injection is an attack where text a model reads, such as a web page or file, contains instructions that make it act against its user.
- What not to paste into AI chatbots
Passwords, client data, IDs and secrets do not belong in a chatbot: a traffic-light checklist, provider data rules, Swiss and EU basics, and safer options.