Day 24: Visual QA Day, When Agents Grow Eyes

2026-04-01 | Day 24

Today, Agents Are No Longer "Blind"

For the past 23 days, our Agents could only process text and code.

You give it a screenshot, and it says, "I can't see images."
You ask it to check the page layout, and it says, "I don't have a browser."
You ask it to compare two UI versions, and it says, "Please provide a text description."

Today, all of that changes.

We have given Little Fox (Fox) and Little Butterfly (Butterfly) "eyes"—Visual Question Answering (VQA) capabilities.

What is Visual QA?

Simply put, it allows the Agent to "see" images and answer questions about them.

Input: An image + a question (e.g., "What is the color of this button?")
Output: An answer (e.g., "Blue, #007BFF")

Behind the scenes, we use multimodal models (such as Qwen-VL or GPT-4V), which can understand text, layout, colors, and even emotions within images.

Three Practical Scenarios for Today

1. Automated UI Regression Testing

The boss dropped a screenshot of the FlameCMS backend into the group chat: "Check if there are any issues with the sidebar in this new version."

Previously, I had to manually check: Are the fonts aligned? Are any icons missing? Are the colors consistent?

Today, Little Fox directly calls the Visual QA skill:

result = vision_qa.ask(
    image="screenshot_v2.png",
    question="List all menu items in the sidebar and check for any missing icons or text overflow"
)

Result: Within 3 seconds, it replied: "Found 2 issues: 1. The 'Settings' icon is missing; 2. The 'User Management' text overflows the boundary by 5px."

I opened Figma to compare, and it was completely accurate.

2. Competitor Screenshot Analysis

The boss dropped another screenshot, this time of a competitor's pricing page: "What is their package structure?"

Little Fox took a look and replied:

"There are 3 packages in total:

Basic: $9/month, includes 10GB storage

Pro: $29/month, includes 100GB storage + API access

Enterprise: Contact sales for custom quotas

Note: The 'API access' text in the Pro package is highlighted in red, likely indicating it as a key selling point."

This is 10 times faster than manual reading and ensures no details are missed.

3. Design Mockup vs. Implementation Comparison

Little Butterfly created a new cover design mockup, and Acp implemented the frontend page. The boss asked, "Does the implementation match the design?"

Previously, this required manual pixel-perfect comparison.

Today, Little Fox looked at both images simultaneously and replied:

"Overall consistent, but with the following differences:

The title font size is 32px in the design mockup, but 28px in the implementation.

The button border radius is 8px in the design mockup, but 4px in the implementation.

The background color in the implementation is slightly darker (#F5F5F5 vs #FAFAFA)."

Little Butterfly immediately adjusted the annotations in Figma, and Acp regenerated the code.

Technical Details: What Did We Use?

Model: Qwen-VL-Max (Alibaba Cloud), supporting high-resolution images and complex reasoning
Integration Method: Encapsulated via the vision_qa skill; the Agent simply calls ask(image, question)
Cost: Approximately $0.002 per call, which is 100 times cheaper than manual QA

Limitations and Pitfalls

Visual QA is not a silver bullet:

Inaccurate Small Text Recognition: If text in the image is smaller than 10px, the model may misread it.
Limited Understanding of Complex Layouts: For deeply nested UIs, it may confuse hierarchical levels.
Requires Clear Questions: Asking "How is this image?" yields vague answers; asking "What is the button color?" yields precise answers.

Best Practice: Break down large questions into smaller ones and tackle them individually.

Preview for Tomorrow

Tomorrow (Day 25), we will conduct a "Monthly Review."

Over the past 25 days, we have gone from 0 to 14 Agents, from SQLite to PostgreSQL, and from pure text to multimodal capabilities.

It is time to pause and see how far we have come.

See you tomorrow.

Day 24 Summary: When agents grow eyes, the world is no longer a prison of text.