Multimodal LLMs: How Vision + Language Actually Work Together

What Is a Multimodal LLM Actually "Seeing"?

If you've used GPT-4o or Claude's image features, have you ever wondered: how does the model connect "image" and "text"? How does it read an error from a screenshot? How does it understand layout from a design mockup?

This article skips the math and explains the core mechanism of vision-language alignment — and why your prompts affect how accurately the model "sees."

Starting with CLIP: The Essence of Cross-Modal Alignment

OpenAI's CLIP, released in 2021, was a major turning point for multimodal AI. The core idea is simple: map both images and text into the same "semantic space," so matching image-text pairs end up closer together in that space.

Training data: 400 million image-caption pairs scraped from the web. The training objective: given a batch of images and texts, bring matching pairs closer, push mismatches apart. This is Contrastive Learning.

# CLIP core idea (pseudocode)
image_embedding = image_encoder(image)     # visual encoder
text_embedding  = text_encoder(text)       # text encoder
similarity = cosine_similarity(image_embedding, text_embedding)
# maximize similarity for positive pairs, minimize for negatives

The result: CLIP's image encoder learned to describe images in semantic concepts, not raw pixels. Instead of memorizing "this cat is Mr. Whiskers," it learned "cat = furry, four-legged, pointy-eared mammal."

How GPT-4o Connects Vision to Language

CLIP is just the first step. The architecture that actually enables large language models to "see and talk" is Visual Instruction Tuning. The core components:

• Vision Encoder (typically ViT): slices the image into patches, converts each patch into a token embedding
• Projector layer: aligns visual embedding dimensions to match the language model's embedding space
• Language model backbone: receives the concatenated sequence of image tokens + text tokens and generates responses

A 1024×1024 image might become 256 patch tokens after ViT processing. These get concatenated with your text tokens and fed to the language model together. The attention mechanism learns which text tokens relate to which image patches.

This is why resolution matters: low resolution = larger patches = lost detail. GPT-4o uses dynamic resolution tiling — higher resolution for detail-rich images like screenshots and documents.

Why the Model Sometimes "Can't See" Your Image

A few hard-won lessons from SFD Lab:

1. Small text in screenshots: If resolution is too low, the model starts guessing characters. Fix: paste the actual code text instead of a screenshot.

2. Complex table images: Table structure is hard to preserve in visual tokens. GPT-4o sometimes swaps columns. Use the PDF interface or OCR-to-text first.

3. Unguided attention: "Look at the whole image" vs. "look at the error message in the top-right corner" produces very different results. Explicitly directing attention significantly improves accuracy.

The Multimodal Landscape in 2026

• GPT-4o: Strongest visual understanding, especially documents, tables, screenshots; real-time voice+vision
• Claude 3.7/4: Excellent code screenshot comprehension, stable long-context image+text handling
• Gemini 2.5 Pro: Best Google Workspace integration, unique long-video understanding (1M token context)
• Qwen-VL: Best for Chinese documents and screenshots, open-source and locally deployable

For local deployment, LLaVA, InternVL2, and MiniCPM-V all run well on MLX with M4 Max — 7B vision models hit 30-50 tokens/s with no problem.

SFD Lab Notes

Our current multimodal use cases: 小刺猬🦔 (QA) sends screenshots to Claude for analysis; 小猎鹰🦅 (security) uses screenshots with text descriptions. Key insight: adding a red-box annotation to screenshots noticeably improved the model's ability to localize problems — exactly how human visual attention guidance works. Multimodal AI isn't magic. Once you understand the mechanism, you know exactly how to use it well.