Rendered at 14:43:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
HarHarVeryFunny 55 minutes ago [-]
The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
MrBuddyCasino 43 minutes ago [-]
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
bicx 14 minutes ago [-]
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
jug 2 minutes ago [-]
I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here without mucking about with sidegrades that I think AI evolution has often felt like lately.
weli 2 hours ago [-]
Anecdotal, opinion:
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
velcrovan 2 hours ago [-]
Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
rib3ye 2 hours ago [-]
anthropic frontend-design skill does a great job with it.
My exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
Segments of the UI that don't conform to any other existing established design or conventions
lelandfe 2 hours ago [-]
areas that look weird
evrimoztamur 2 hours ago [-]
Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.
ParanoidShroom 22 minutes ago [-]
I run the free service https://countrx.app/ so i have some idea what goes into counting.
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong.
I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools?
Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
fpgaminer 28 minutes ago [-]
Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
cdolan 10 minutes ago [-]
Luna is pretty strong as well. been using it for projects the last two weeks and its strong
mv4 1 hours ago [-]
Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
maxime_cb 57 minutes ago [-]
I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)
https://ieeexplore.ieee.org/document/1674847
G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977.
DOI: 10.1109/TC.1977.1674847
lebek 54 minutes ago [-]
The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
kzrdude 2 hours ago [-]
In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
defrim 48 minutes ago [-]
Seems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected
bearjaws 45 minutes ago [-]
It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
repeekad 24 minutes ago [-]
How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
criddell 27 minutes ago [-]
Are any of these vision benchmarks binocular in order to introduce depth perception?
I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
WarmWash 49 minutes ago [-]
It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
kherud 49 minutes ago [-]
So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.
prathje 1 hours ago [-]
I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...
I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).
5555watch 1 hours ago [-]
All of your use cases are very advanced.
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
chasd00 1 hours ago [-]
One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.
Mashimo 55 minutes ago [-]
This article is about vision, not image output.
stavros 8 minutes ago [-]
Hence the "catch and fix mistakes" part.
25 minutes ago [-]
iamniels 1 hours ago [-]
I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
rhplus 36 minutes ago [-]
Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
throwup238 1 hours ago [-]
Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
adroitboss 1 hours ago [-]
I didn't expect Gemini 3.5 Flash to top basically every metric in this article.
SweetSoftPillow 1 hours ago [-]
In my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.
WarmWash 45 minutes ago [-]
Gemini has long been the vision champion, but there aren't many benchmarks and coding is where all the hype is.
Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
LollipopYakuza 1 hours ago [-]
Same. I scrolled back up to see if I read the title correctly. It's important to note that it is the best... OpenAI released. Not the best overall.
comboy 1 hours ago [-]
Does any popular NVR make a good use of LLMs (especially local models) getting decent at vision?
trumbitta2 1 hours ago [-]
"Best iPhone ever" vibes.
sscaryterry 2 hours ago [-]
My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.
howdareme 2 hours ago [-]
How can a vision model have taste?
sarreph 2 hours ago [-]
If you're doing any kind of inference that is multi-modal and non-factual, opinions and biases will affect any kind of assessment of a visual that you provide to a model.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
DaiPlusPlus 2 hours ago [-]
> if the interface elements have strong information hierarchy
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
sarreph 6 minutes ago [-]
I disagree.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
sscaryterry 2 hours ago [-]
Replace taste with consistent if that helps you. Can it follow a design system...
yreg 2 hours ago [-]
As a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.)
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
sscaryterry 2 hours ago [-]
This is not my experience at all.
velcrovan 2 hours ago [-]
So, formulaic output…the opposite of taste
sscaryterry 2 hours ago [-]
Not really. Compliance with the letter of the law doesn't mean the intent is complied with.
RugnirViking 42 minutes ago [-]
It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure
logicallee 1 hours ago [-]
I agree. It did very well on an extremely challenging task.
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
Razengan 2 hours ago [-]
For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
dev_hugepages 1 hours ago [-]
I'm unsure why you're using an LLM to generate images. Don't we already have models (some made by the same company) that do this?
sscaryterry 2 hours ago [-]
> it constantly keeps getting something wrong no matter what I tell it
This 100%
thatcat 2 hours ago [-]
did you try segmenting it first?
Razengan 17 minutes ago [-]
At first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like.
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
catigula 1 hours ago [-]
Still not quite as good as gemini.
iamleppert 1 hours ago [-]
Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.
ImageXav 6 minutes ago [-]
Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.
GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.
Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at:
> create polished code
Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC?
> high-impact animations
That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes.
> generic system fonts, predictable purple gradients, and cookie-cutter components.
This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory.
> thoughtful typography with unexpected font pairings
The above fragment set my alarm-bells off. Yikes.
> scroll-triggered interactions
Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible.
------
The SKILL.md itself[3] grinds my gears too:
> Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's.
Claude has no way of knowing what designs are actually unique or not...
> For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment
...this is exactly what everyone else's web-pages look like!
> For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns
...I called this out weeks ago[4], lol.
and I could go on. This is all quite painful to read.
------
[1] https://claude.com/plugins/frontend-design
[2] https://github.com/anthropics/claude-plugins-official/tree/m...
[3] https://github.com/anthropics/claude-plugins-official/blob/2...
[4] https://news.ycombinator.com/item?id=49187385
The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.
Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.
Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.
But for being able to one shot with this accuracy... IMPRESSIVE
https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847
In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.
I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.
I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.
I will add that all frontier models were fine with such tasks from the early 2024's.
In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
For example, a UI / UX professional being asked to appraise a website screenshot may determine that the image in question has "desirable" traits which are inherently not deterministically measurable. Such as, if the interface elements have strong information hierarchy, or if they are deemed to be "fashionable" with current UI trends.
...but that's an example of a UX/usability matter that can be assessed objectively and non-subjectively.
Is 16 px or 14 px a better font-size value for a subheading, in a hypothetical layout? Immediately that kind of decision, where both options are objectively good for 12 px paragraph text, suddenly becomes an issue of taste that cannot be evaluated crudely by an algorithm.
But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6.
Of course only if the design is achievable in the design system.
I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.
In addition, the poster itself also happened to contain similar clothing.
You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...
While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.
An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
This 100%
The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes.
So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.