从像素到文字：视觉语言模型究竟如何工作

Artificial Intelligence Machine Learning Computer Vision Deep Learning LLM From Pixels to Words: How Vision Language Models Actually Work How vision encoders, projection layers, and language models work together to understand images Rithikha S 7 min read · Just now -- Listen Share You upload a photo and ask: “What is happening in this image?” A normal AI chatbot would ignore the image and ask you to describe it in words. But a Vision LLM actually processes the photograph. It converts the visual information into something its internal language model can reason about, combines that with your text prompt, and generates a response. The question seems simple: How does an AI model turn pixels into words? The answer is more interesting than you might think. What Is a Vision LLM? A Vision LLM is a large language model that can accept both an image and text as input and respond in text. Think of it as a chatbot that can also see. You’ve probably used this already if you’ve asked ChatGPT, Gemini, or Claude questions about uploaded photos, charts, or screenshots. The technology powers features like photo Q&A, document extraction, and chart analysis. But Vision LLMs aren’t simply regular language models that somehow learned to “see.” They’re a carefully constructed system that connects visual information with language in a specific way. Why Do We Need Vision LLMs? The world contains far more visual information than what is written as text. Think about everything around you: photographs, charts, receipts, screenshots, scanned forms, diagrams, handwritten notes, and physical surroundings. A text-only language model can’t directly interact with any of this. A text-only LLM can’t: - Interpret a chart in a report - Spot a defect in a photograph - Read a scanned form or receipt - Describe a scene for someone who is visually impaired A Vision LLM can potentially do all of these things — not perfectly, but meaningfully. The limitation of text-only models created an obvious problem. If most of human information is visual, then an AI model that ignores images is working with one hand tied behind its back. How Is a Vision LLM Built? A Vision LLM has three main components that work together. Think of them as an eye, a translator, and a brain. The Vision Encoder (The Eye) The vision encoder processes the raw image and converts visual information into numerical representations. A common choice is the Vision Transformer (ViT), which treats images as a sequence of small patches, similar to how language models treat text as a sequence of words. The Projector (The Translator) Here’s the key insight: the vision encoder produces numerical representations in its own “space.” The language model expects representations in a different format. The projector bridges this gap. Imagine a foreign film where one person watches the screen and another person reads a subtitle track. The vision encoder “watches.” The projector translates. The language model “reads the subtitles” and understands the plot. The Language Model (The Brain) The language model receives visual information together with the user’s text prompt and generates the response. The language model itself never processes raw pixels directly. It works with the translated visual representations that the projector provides. The flow is simple but crucial: Image → Vision Encoder → Projector → Language Model → Answer Press enter or click to view image in full size *A simplified architecture of a Vision LLM. The vision encoder processes the image, the projector translates it, and the language model generates the answer. Source: Image by the author.* From Pixels to Words: Step by Step Let me walk through the four-step process that transforms an image into a response. Step 1: Cut the image into patches Imagine taking a photograph and overlaying a grid on top of it. Then cut the image along the grid lines. You’ve divided it into small square patches, like a puzzle. Step 2: Convert patches into numbers Each patch becomes a vector — a list of numbers that captures the visual content of that region. Where there’s red, the numbers reflect that. Where there’s texture, the numbers capture it. Each patch is now a numerical representation. Step 3: Translate for the language model The projector takes these numerical representations and reshapes them. It’s not changing what the information means. It’s changing the format so the language model can work with it, similar to translating spoken words into written text. Step 4: The language model reasons and replies Now the language model has two inputs: the visual tokens from the projector and your text prompt. It combines these and generates an answer. Get Rithikha S’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in The analogy is useful: you’re giving the language model a translated transcript of everything important in the image, and asking it to discuss it. Press enter or click to view image in full size *The four-step process: Images are divided into patches, converted to numerical embeddings, translated by the projector, and combined with text for the language model to process. Source: Image by the author.* Three Terms You Need to Know Vision Transformer (ViT) The most common vision encoder architecture. It processes images by dividing them into patches and treating those patches the way a text model treats words. Embedding or Token A numerical representation of a piece of information. In text, an embedding might represent a word. In Vision LLMs, tokens can represent image patches or visual features. These are the “numbers” flowing through the system. Projection Layer A relatively small neural network that maps visual embeddings into the same representation space as text embeddings. It’s the translator. How Are Vision LLMs Trained? Training happens in two distinct stages. Stage 1: Learning to Align The model learns to connect images with their descriptions. It’s shown millions of image-caption pairs: - Image of a dog + “A dog running on grass” - Image of a car + “A red sports car” - Image of a market + “People at a market” Here’s what’s important: the vision encoder and language model are usually already pretrained. Only the projector is trained during this stage. The model learns to adjust the projector until the visual and language representations align — until the “translation” is accurate. Stage 2: Instruction Tuning Now the model learns to actually follow instructions and have conversations about images. It’s trained on examples like: - “What color is the car?” → “Red.” - “Summarize this chart for me.” - Follow-up questions about the same image. This stage teaches the model how to be useful in actual conversations, not just how to associate images with captions. Press enter or click to view image in full size *Vision LLM training happens in two distinct stages. Stage 1 learns to align images with language. Stage 2 learns to follow instructions and hold useful conversations. Source: Image by the author.* Training Data: Where It Comes From Vision LLMs learn from four categories of data: 1. Web images and captions — Photos paired with alt-text or descriptions scraped from the internet. 2. Research datasets — Large public collections already labeled and organized for AI research. 3. Human-written Q&A — People writing sample questions and answers about specific images. 4. Documents and charts — Screenshots, tables, and graphs so the model can learn to read them. The quality and diversity of training data matters enormously. Poor or biased data leads to biased results. This is a known limitation worth keeping in mind. Real-World Applications Today Vision LLMs are already deployed in practical contexts: - Photo Q&A: Upload an image and ask questions about it - Accessibility: Describe visual surroundings aloud for visually impaired users - Document reading: Extract information from receipts, forms, and IDs - Chart analysis: Summarize graphs or spreadsheet screenshots instantly - Self-driving systems: Understand road scenes, signs, and pedestrians - Content moderation: Flag unsafe or policy-breaking images automatically In each case, the ability to process visual information directly, without a human converting it to text first, creates genuine value. What Vision LLMs Still Get Wrong Vision LLMs are powerful but imperfect. Hallucination: The model sometimes describes objects or details that aren’t actually present in the image. It confidently describes something it’s never seen. Weak spatial reasoning: It can struggle with precise tasks like counting objects, judging exact distances, or determining relationships between items. High compute cost: Processing images requires significantly more computational resources than processing text alone. This has real cost and latency implications. Bias: Training data reflects stereotypes and gaps present in the internet. These biases can influence how the model interprets images. A Vision LLM being able to “see” doesn’t mean it sees perfectly. The limitations are real and worth understanding before relying on these systems for critical decisions. The Bridge Between Vision and Language A Vision LLM doesn’t magically teach a language model how to see. Instead, it builds a bridge between visual information and language. The architecture is elegant: an encoder that specializes in vision, a projector that translates, and a language model that reasons. Each component does one job well. Pixels → Vision Encoder → Embeddings → Projector → Language Model → Words The interesting part isn’t that AI can now look at an image. The interesting part is that pixels can now become part of the conversation — that visual information can flow into the same reasoning engine that processes text, with all the continuity and context that implies. That shift from “AI that only reads text” to “AI that understands multiple types of information” is what makes Vision LLMs worth paying attention to.