Rendered at 16:13:22 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
CMay 3 days ago [-]
Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it.
Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks.
The VRAM usage seems way less efficient than Gemma 4 or Glimmer though, with 32K of context taking 2.5GB of VRAM. With those, even with MTP or a DFlash model loaded, you could still fit 256k-768k of context. With Qwen 3.8 27B I can't even fit 128k if I quantize V to Q4_0. Maybe with some trial and error I can find some settings that perform well enough with a larger context window that it's still useful for longer tasks.
Lots more testing to do, though I was getting some decent results out of Muse Glimmer which was more than twice as fast and supported huge context windows, managing to solve some bugs that Gemma 4 struggled with. I can't even begin to throw that task at Qwen, because just the prompt alone would use the entire context window and then it would reason for probably that same amount.
If you've got a 32GB card, it should be a decent model even if it really is memory hungry.
EDIT: Tried a few kv cache quantization settings, but it failed with those. I designed this benchmark to be pretty brutal in the face of KLD and any reasoning quality loss, so it's not too surprising. Gemma 4's QAT held up pretty well, at least and could consistently complete it.
pulse7 2 days ago [-]
Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B?
What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B?
I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...
If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario.
My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16.
Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all.
For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used:
llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16
jodleif 2 days ago [-]
Then you might be missing SWA. Gemma models are extremely memory hungry without
CMay 2 days ago [-]
So long as they have flash attention enabled, Llama.cpp enables Sliding Window Attention by default for Gemma 4 models. Even if they're using Ollama or LM Studio I would expect those to mostly be doing the right things.
DiabloD3 2 days ago [-]
I would not expect Ollama to be doing the right thing fwiw.
nobrains 2 days ago [-]
"correctly reason through one of my private benchmarks"
i would also like to make one myself for my testing. could you give a rough idea or an outline or point in the general direction on what to do?
Qwen’s 3.6/3.8 27b actually has some algorithmic advantage when it comes to the kv cache size needed, so it actually needs less memory at equivalent context. My experience using both in vllm supports, with considerably more overhead in context size on these models than Gemma 4 31b, and better performance in most tasks I’ve tried on both models.
lhl 2 days ago [-]
Qwen 3.x does have an advantage but it's relatively small (64KB/token vs 80KB/token) - Gemma4 actually has less % of full attention layers, but the largest geometry and has the biggest "fixed" state for it's non-global layers. Muse Glimmer actually has by far the lowest per-token cache usage for the competitive 30B-class dense models - it's at about 13KB/token - very aggressive GQA (32Q/2KV) and also by far the smallest QKV dimensions.
Actually perf (speed) is going to mostly on token output, and here Qwen 3.x historically tends to lose badly as it tends to overthink a lot. I'll be running evals on 3.8 myself this weekend to see how its reasoning levels perform.
I was quite impressed by Muse Glimmer, and while I am sure people will observe that it is less good on benchmarks, my first experiences with this new 27B have been somewhat exasperating, whereas testing Muse Glimmer was rather fun. I have not tested either in an agentic context, mind you.
CMay 3 days ago [-]
Yeah, Glimmer is excellent. You don't really test Glimmer with one-shots, because it's explicitly designed for multi-turn solution finding. The way I see it, if I've got a task that could be done either agentic or requires a lot of context (for example, dumping 600KB of API documentation and another 300KB of codebase for a project) then I would reach for Glimmer easy and it seems like it could get there most of the time.
Qwen might be useful to bring out for a second opinion on some more focused details that are largely information complete. Like, use Glimmer to bring together all the relevant critical data and evaluate what the actual problems are, then maybe prototype a solution. If it's still acting up, maybe throw the resulting context at Qwen and let it meditate on it.
I think there was some study done where ideally you would want to throw a bunch of different models at a problem since they don't all have the same perspective or diagnosis on what the problems or the solutions are.
bitexploder 2 days ago [-]
That is exactly how this model has worked for me so far. Muse on a one-shot task will get to 80%. And if you even nudge it and say, "Hey, finish up," or "Review the syntax," boom, it's done. And I'm getting 20 t/s with Ollama on a MacBook M5 Pro with 48GB of RAM. It is a seriously impressive little model.
pram 3 days ago [-]
Glimmer works really well as an "explore" agent model (like in Opencode.) It seems to be extremely efficient at searching and collating that info, and executing commands.
From my testing so far, Qwen 3.8 is better at code but it tends to meander and take forever if it has to look in a lot of places. Glimmer will use like ~1k tokens to formulate a plan and Qwen 3.8 will routinely go over 10k
dofm 2 days ago [-]
Have you tried turning down the new Qwen's reasoning effort level from xhigh, which it defaults at?
LM Studio isn't exposing a dropdown for this, at least with the unsloth build.
Unsloth Studio / Desktop does.
pram 2 days ago [-]
These templates actually fix the effort selection for LM Studio/3.8
Oh, that's cool, I saw those and I did wonder! Thank you.
nullc 3 days ago [-]
I wouldn't find glimmer interesting except that it has much less memory usage per token of KV than Qwen. So I can get 24x concurrent glimmer on 2xRTXA6000 (with 128k context) where I can only get 6 Qwen 27b. This means I can get something like 4x the aggregate tokens/s out of glimmer.
For some usages that speedup more than makes up for it being inferior to Qwen intelligence wise.
chr15m 3 days ago [-]
Glimmer is fun because it's fast, tight, and doesn't wander or waffle. My favourite local model so far.
embedding-shape 3 days ago [-]
> The VRAM usage seems way less efficient than Gemma 4 or Glimmer though
Maybe it's implicit that you're using llama.cpp (although you don't mention GGUF), but it's hard to reach concrete conclusions about the model architecture based on one implementation in one runtime.
apitman 3 days ago [-]
Aren't things like KV size inherent to the model?
TomatoCo 3 days ago [-]
There's a tiny bit of play, like sliding window attention. As tokens leave the sliding window you can keep them or discard them. If you keep them, you can freely truncate the context and resume generation from an earlier point. If you discard them, you have to recompute the KV cache up to that point.
Llama.cpp checkpoints and moves snapshots of the cache to main RAM for faster resumption after truncation.
monster_truck 2 days ago [-]
In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without
What exactly are you doing that the prompt is eating an entire 65536 window? Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing. This allows it to keep the initial instructions in the start of the window and toss out the middle as it goes. IME many people who have written off local models entirely are, for lack of a better term "not holding them right" and consider them worthless.
Definitely also try IQ4_NL for K/V if you haven't. Because it's non linear it's far more hit/miss from model to model and especially quant to quant, I've found generally that it shines brightest when you start with a Q6K+ quant that you otherwise wouldn't bother with because of its size, which it then makes up for in both inference speed and often a larger context.
I do agree about Glimmer, though. It is quite good, far better than the benchmarks let on, especially in heavily agentic cases where it needs to rampage around the OS and utilize many different utilities to zero in on things. It is especially good at being told to try something itself, and if/when it fails, try Qwen, and if Qwen can't do it, call out to Deepseek.
CMay 2 days ago [-]
> In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge
If you set manual MTP settings, you'll override dynamic adjustments the inference engine will try to do. Sometimes the dynamic adjustments aren't optimal. With the settings I use, MTP is always a net win.
> What exactly are you doing that the prompt is eating an entire 65536 window?
I'm not using the full context window.
> Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing.
Tools would not help.
searealist 2 days ago [-]
> In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without
Speaking in terms of wall clock, the expensive part of decode is fetching the weights from memory. Predicting and validating a bunch of tokens using the already fetched weights is insignificant in comparison. Even if you have a poor acceptance rate for predictions, you won't really see a slowdown vs not using MTP.
bitexploder 2 days ago [-]
Have you tried Muse 30B yet? I have been impressed with it. I have Qwen 3.8 27B hammering away right now against Muse. And Muse is doing a little bit better.
imagetic 2 days ago [-]
Vibes
arcrs 13 hours ago [-]
this is efficiency im looking for. artificialanalysis.ai model review not up. so considering output token per intelligence, do u think is it better than muse glimmer or no?
bmitc 3 days ago [-]
> correctly reason through one of my private benchmarks
Want to say more about these private benchmarks? :)
Retro_Dev 2 days ago [-]
seems to me like "private" is a good descriptor - I also have a set of "private" test cases - and they are kept private on purpose so they aren't scraped and fine-tuned on.
dofm 1 days ago [-]
I have a sneaking suspicion that Qwen is fine-tuned on youtuber test cases (like Luke's Dev Lab, where Qwen 3.8 27B has just done almost eerily well).
Part of my suspicion is drawn from the thinking trace I got when I tested the car wash problem. That really does seem to have been post-trained; it's too good.
e.g. Gemma 4 26B solves this concisely without adding any filler about fuel economy or how long it will take, but it generally gets there by breaking down the problem in the thinking trace the way you'd expect.
Qwen 3.8 27B is just a little too certain right off the bat in low reasoning mode.
bmitc 2 days ago [-]
That clarifies it.
urbnspacecowboy 3 days ago [-]
> Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks.
I don't expect you to blab publicly about your private benchmark, but what sorts of reasoning does it require?
CMay 3 days ago [-]
Well, I will say:
#1: it does not require deep world knowledge, because that's not what local models are for.
#2: it directly attacks drive-by understanding, overly linear processing training, poor attention mechanisms, poor reasoning patterns or lazy assumptions that ignore very easy low hanging fruit.
#3: it requires solid instruction following in the face of errors. a lot of models will run into errors and then fall back into some kind of error recovery process that bypasses instruction following.
#4: does not require prompt fine tuning to tweak to each individual model. they all seem to understand.
#5: not unfair. almost every model demonstrates in their reasoning that they have the necessary information that if reasoned about appropriately, could arrive at the correct answer.
#6: not designed to add unnecessary complication that it is intended to exhaust reasoning budgets of any sort, so it is not inherently unfair to models that reason a little more or less. for example, it does not require unnecessary reasoning soaks (ie: hiding the prompt inside base-64 encoding)
#7: has real world use and is probably applicable to overall ability to generalize.
#8: can be scaled up as models get better.
#9: is a very good indicator of how bad a model is falling apart under various inference settings.
akoboldfrying 2 days ago [-]
I especially like #8. If you have some free time (don't we all have so much of that?) it would be really interesting to run a binary search on each model you have, to see at what size/complexity level it manages to solve the problem, say, 50% of the time.
m-s-y 2 days ago [-]
Are you willing to share this benchmark’s internals? Kinda weird to expect folks to take you at your word without the ability to “trust but verify”
I'm not asking anyone to take my word, they can believe or not and in practice people should be taking signals from a variety of places and doing their own testing to see how models behave in their own use cases. What I'm measuring and why I'm measuring it may not be the most important metric for your specific use case.
Most other models are simply failing at these tasks. I think the tasks are relevant to overall model capability, but they are not the only metric. You don't give a jellyfish a tool and expect it to produce wonders, so the other capabilities of the model matter.
ziofill 2 days ago [-]
How much time did you invest in creating this benchmark? Any recommendations/resources you could give on how to do it?
crypto137 3 days ago [-]
It writes turing complete Beauty and the Beast fanfic.
jamiejquinn 3 days ago [-]
I laughed so hard at that, thanks
m00dy 2 days ago [-]
:D, just upvoting this in case of someone downvotes
Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.
(No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a small chainring.")
It took 21 minutes(!) and used 22,276 reasoning tokens to produce 3,223 tokens of output.
(For the "they're training on your benchmark now" crowd, all of that cheating didn't prevent it from spending 20 minutes thinking about the task first! You can see the reasoning trace in the link I shared.)
For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model.
Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right.
It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
oarsinsync 2 days ago [-]
Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike.
I suspect if you showed one human a million photos of a bike they might get it right.
fwipsy 2 days ago [-]
I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks.
Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.
*Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
ElectricalUnion 2 days ago [-]
AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.
Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
walrus01 3 days ago [-]
It is interesting that it went with something like:
"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"
I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
Einenlum 3 days ago [-]
Damn
That's very good
zkmon 2 days ago [-]
>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.
How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
anakaine 2 days ago [-]
Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other.
The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.
Also:
> Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.
theplumber 3 days ago [-]
Gemini is worse
lacoolj 3 days ago [-]
Holy crap that is above and beyond
But, not animated like the Qwen 2.4T one you showed, so minus points for that
irthomasthomas 2 days ago [-]
Animation was not requested.
lacoolj 3 days ago [-]
Totally agree. I actually didn't know you already got to it and I ran it on mine. It even added a scarf lol
Took almost 90 minutes on my Threadripper Pro, and feels well worth it.
28k tokens from the Q8 model
HanClinto 3 days ago [-]
And yours got the chain!
walrus01 3 days ago [-]
What's the result if you ask it for a "pelican equipment case" ? I've been trying the anti-bicycle pelican on some LLMs and the results are much more varied than the bicycle prompt. Some have very different ideas of pelicans (you'll get a small case with one DSLR camera in it, or a long rifle case, etc). You'll also get cases that are isometric view, or flat plane view from the front, or open or closed.
This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working.
llm -m lmstudio/qwen/qwen3.8-27b 'Generate an SVG of a pelican equipment case'
walrus01 3 days ago [-]
That is pretty good! It's interesting it chose orange, because that's actually a somewhat rare color for pelican cases. I would say 95% of the ones going around the world right now are either black or FDE (flat dark earth) tan colored. There are some orange ones, for sure, if you google image search "orange pelican case", people use them for emergency/first aid supplies or electronics. But only the smallest ones are available in orange.
It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.
pbhjpbhj 3 days ago [-]
What are the chances that the colouring of actual pelicans is bleeding in to the prompt. Seems it could be that?
walrus01 3 days ago [-]
Doubtful? Without knowing anything about how it was trained, if you google image search "pelican bird" they are predominantly white in color. Though the bill is certainly orange, different shades depending on camera and lighting conditions. The SVG attempt looks more like an attempt at the product photos if you search "pelican case orange".
ethbr1 3 days ago [-]
Need more tests of 'pelican bird standing in pelican equipment case' ;)
walrus01 3 days ago [-]
Or to make it even more tricky, since Pelican cases will float when closed, something like "generate an SVG file of a Pelican bird floating in the ocean next to a Pelican case".
fragmede 3 days ago [-]
I've been getting ads in my Instagram for http://colorcase.com. They add colors and wheels to cases and I'm sharing them because you're right, most of them aren't orange. But they could be!
walrus01 3 days ago [-]
Wow, those are expensive. I've been buying pelican 1510 and similar from eBay and b&h used/refurb. The harbor freight pelican knock off that is their 3800 and 4800 product numbers are also not bad. It's like 85% as good as a real pelican for 40% of the price.
fragmede 2 days ago [-]
it's just money and you wanna look good. You're worth it.
wincy 3 days ago [-]
Not too bad, on my 5090 I got ~160tokens/sec and it took 3m20s to generate a pelican using Ninfer which someone else mentioned. I don't know why I didn't get any sweet sweet seagulls in my background, even on xhigh reasoning effort.
I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4.
Some will say conflating flying and flying a kite is bad, but I like it.
frumiousirc 2 days ago [-]
> Bicycle is the right shape. Pelican beak is excellent. Nice background.
Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube).
Consider a series of followup benchmarks.
With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion.
Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark.
Last phase is to iterate with the goal to drive the number of findings to zero.
kamranjon 3 days ago [-]
Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.
gerdesj 3 days ago [-]
I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format.
I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted.
My word! It looks rather decent. Much better than previous efforts.
Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though.
Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).
redox99 3 days ago [-]
It's definitely trained on generating SVGs. Not that they trained on the pelican prompt itself, but probably a more general RL step that has it generate SVG and a stronger vision model judges it.
It's up to you if that counts as benchmaxxed.
nl 3 days ago [-]
Generation of SVGs is (obviously) a generally useful task.
maldie 2 days ago [-]
Even with "low" thinking effort and Q4 it makes reasonable pelican, only 4k tokens and half a minute on 4090: https://imgur.com/a/yd5rNCp
KerrAvon 2 days ago [-]
If I'm doing the math right, that's like 17 t/s? I haven't played with Qwen 3.8 yet, but that seems really slow for a 27B on an M5 Max with sufficient RAM to hold it in memory.
latentsea 2 days ago [-]
I asked it to make an animated svg of a pelican doing tricks on a bmx bike and it did. I then asked it to turn that into a browser game. This model fucks.
distrill 3 days ago [-]
this sort of test surely only works when it's a brand new concept/prompt. like the exact same models trained newer data will surely get better at putting this together over time
notnullorvoid 3 days ago [-]
Try it out with your own prompt, I suspect you'll be surprised.
nullbio 2 days ago [-]
Ask it to do a zebra on a jetski to escape the benchmax.
sahadi 3 days ago [-]
what type of laptop runs something like that? using mac or egpu?
delduca 3 days ago [-]
See, like my comment before. We don’t need the biggest/highest/frotiest model.
geniium 3 days ago [-]
I think the model got trained only on pelican to get there
walrus01 3 days ago [-]
We should start asking it for an Albatross instead.
dexterlagan 2 days ago [-]
Tested the model briefly with my usual eval: a couple of questions on general knowledge most small models often get wrong, then write a fully-featured todo list app in JS, then rewrite the same app in Rust with Tauri. Granted, most models are well trained on basic todo apps, but it gives me an idea of the basic SWE capabilities I can build on. As far as I'm concerned, if it can successfully setup a local git repo, write a todo list app skeleton that works, I can work with it.
SWE: model is strong for its size. It one-shotted the Web app, had no bug. The Rust rewrite only had one bug (reordering didn't work immediately - fixed in one prompt). Committed locally then pushed to my GitHub (https://github.com/DexterLagan/RusTODO). Can't complain. If it can do that reliably, I can use it to make whatever I need.
General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.
Performance on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Perfectly usable for local coding through OpenCode.
Verdict: very nice local and free backup to my usual GPT/Claude/DeepSeek for code. Good for Web searches via Brave search tool calls. What more do you want from a small local model?
cjbprime 2 days ago [-]
> General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.
I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.
Yajirobe 2 days ago [-]
Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
ChucklsTheBeard 1 hours ago [-]
Ideally it would run a web search to check questionable knowledge. Google-fu has been one of the most important SWE skills for a long time!
dexterlagan 2 days ago [-]
Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.
croes 2 days ago [-]
Models don’t know that they don’t know.
mdp2021 2 days ago [-]
> know that they don’t know
And we are waiting for architectures that do - because it's duly.
Zambyte 1 days ago [-]
Honestly it seems like a job for the harness, rather than the model. Sample the model with the same question, perhaps with varying temperature (?), and use that to establish a degree of confidence in the answer. If the model provides very different answers every time, respond that it doesn't know. If it responds with the same answer usually but a different answer sometimes, respond with moderate confidence. If the model always responds with the same answer, respond with certainty.
mdp2021 1 days ago [-]
If we want to implement Intelligence, and especially now that "the box is open" we must, we can play with the "intuitive" LLM architecture to understand it and squeeze it to its potential yeld, but at some stage we have to actually implement intelligence. That implies notions of confidence and a Foundational Theory of Knowledge (knowing why you know something), among the rest (one shot learning, update through reflection etc.).
jychang 1 days ago [-]
that could work sometimes but that's terribly hacky engineering
mbmbn 2 days ago [-]
[dead]
kamranjon 2 days ago [-]
I actually disagree that this doesn’t mean anything. I understand the contention that it’s not measuring the quality of the model in general, but I think it is measuring something useful.
A good example of this is planning hardware projects - a larger 200b plus model like DeepSeek V4 flash will recommend parts like motors, real time clocks, voltage regulators etc and it will do so providing exact model names and specifications.
I wouldn’t expect a smaller model to encode all of this information, but it is helpful to understand where that cutoff is because it changes what the model might be useful for. It is a very crude way of measuring because it comes down to the balance of training data at sizes this small - but I do think it conveys something that is helpful in real world tasks.
dexterlagan 2 days ago [-]
Agreed. It's just to see how much knowledge the model still has for its size. For comparison, no other model of this size has ever given me the correct answer to the Keel question. Even GPT-OSS-120B got it wrong, but it's expected for its relative old age.
ra 2 days ago [-]
I concur, treating models as question and answer machines and judging them on recall is meaningless, unless you're measuring quantisation impact on a foundation model maybe.
dexterlagan 2 days ago [-]
100% agree, it's really just for kicks&giggles. The fact that the model answered correctly, unlike every other model of its size before it, still pleasantly surprised me.
pkghost 2 days ago [-]
> on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode.
Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!
terhechte 2 days ago [-]
Which setup did you use? MLX/GGUF, Quant, Engine (e.g. llama.cpp or MTPLX, etc)? There’s so much variety these days.
dexterlagan 2 days ago [-]
It was in LMStudio (llama.cpp), Q4 by Unsloth. Applied the recommended defaults published by Unsloth.
dexterlagan 2 days ago [-]
Meaningless yet fun fact: DeepSeek V4 Pro 0813 made a much worse icon for the same app, and only produced an SVG I had to convert manually to .png. Qwen3.8 made a perfect icon in .png. I don't yet know how it did it, but it did it.
Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.
NamlchakKhandro 2 days ago [-]
copilot isn't a model. it's a model hoster.
swores 2 days ago [-]
> write a fully-featured todo list app in JS
Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).
dofm 3 days ago [-]
There's a real change (compared to 3.6) in the way it writes in thinking — it drops words like "to" and "we" in "We need to", talks generally in note form, drops the/and all over the place, avoids "for".
"Need be helpful concise", "Need maybe not overdo", "Need ask!" Almost caveman.
I have an (unsourced, vague) suspicion that this rather unique thinking trace pattern is actually hobbling the MTP predictions, which seem to perform poorly.
Other notes: it uses the trick of repeating the prompt in the thinking trace.
It also worries about hidden chain of thought appearing in the final answer. It talks about "desired oververbosity 9", which is new. A bit GPT-ish.
It is being extraordinarily thorough in thinking through one of my code requests, but I don't know if the net result will be any better than the 35B MoE.
I asked it to ask me clarifying questions — it did, and it offered me a list of defaults I could simply agree to.
I don't think it is necessarily overthinking in the looping sense, but it is in the being exhaustive sense. I need to explore how it does with a tighter reasoning budget.
I am impressed but I am definitely in Camp Please-35B-A3B-When? here, because on an M1 Max this isn't really practical. I hope they do one, though I think they may not.
WhyNotHugo 2 days ago [-]
This is pretty much Chinese grammar with English words. In Chinese you'd say "need ask" or "I need ask" if you want to be unusually verbose, but there's no "to".
All your examples align with this too. It just sounds to me like its optimising by removing the superfluous English words and thinking more Chinesely.
jchook 2 days ago [-]
After learning a little Mandarin I was amazed at how much simpler the Chinese grammar is than all Latin, Germanic, Romance, etc languages.
No gender, cases, tenses, conjugation, articles, word order inversion, etc. It's remarkable.
Mandarin has it's own complexities but grammar wise, it feels more optimized or simpler.
qeternity 2 days ago [-]
Has nothing to do with Chinese.
Frontier labs have already been doing this for a while, verified in smuggled traces from OAT/Ant.
I also noticed that if you ask it to write text content to a certain word length (like "please write 250 words about the potato"), it now takes its sweet time "manually" counting each word in its draft output. It puts a number next to each word and iterates through all of its draft paragraphs and tries to stick to a strict word count. This is with temperature 1.0 as recommended.
ricardobeat 3 days ago [-]
This has been common in Kimi K3, and now DS Flash/Pro as well. Evidence of cross-training?
dofm 3 days ago [-]
It looks like the caveman speak is a facet of the 'xhigh' reasoning effort. Set (bodged!) to medium it is not caveman.
dofm 3 days ago [-]
Annnnd the code of my WP code test is not better. It is bushy, overcomplicated, and has gone around the houses to do stuff it would not need to do if it hadn't overthought.
Oh dear. I need to try to understand what is going on here.
eek2121 3 days ago [-]
thinking is set to max by default. I bet that turning it down would solve this.
dofm 3 days ago [-]
I think so too — it is something to test, for sure.
ETA: a bit of testing before I climb the wooden hill to Bedfordshire.
LM Studio doesn't seem to display the little dropdown to set reasoning effort, so I bodged the chat template on load to get it to choose 'medium'.
As soon as you switch away from xhigh, it goes back to thinking in normal sentences like Qwen 3.6, rather than in sort of quasi caveman.
And you get all the Wait, Actually, No wait… stuff back.
And it is behaving a lot more like it used to. So that is pretty interesting.
airbreather 2 days ago [-]
Yeah well for whatever reason I often tend to drop a lot of those small words when typing something on keyboard...
satvikpendem 3 days ago [-]
As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc.
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs?
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
suprjami 3 days ago [-]
You have understood correctly.
One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.
But they don't.
Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.
I also don't understand how this repeatedly happens.
z4y5f3 2 days ago [-]
I did SFT / RL post-training on Qwen3 models a bit. This is an issue that dates back long ago.
My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.
satvikpendem 3 days ago [-]
Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.
kzrdude 3 days ago [-]
Is the chat template used at all when they benchmark the model?
dannyw 3 days ago [-]
It’s unlikely they are benchmarking that downstream. They probably have private benchmark scripts with purpose-specific run telemetry and logging, etc.
I’m sure there is some basic testing but it might be agentic (LLM likes its own output) and maybe just some human smoke tests.
kzrdude 2 days ago [-]
So what's needed to solve this is that someone publicly and popularly benchmarks using the chat template. So that they have an incentive to look good using it.
runeblaze 3 days ago [-]
they likely use their internal infra to run benchmarks; aligning external releases with internal environments is always painful and somewhat underincentivized
hedgehog 3 days ago [-]
Yes, I was fixing issues piecemeal until I found the froggeric template, I've had to fix I think one issues with that one but it's better.
alfiedotwtf 3 days ago [-]
The chat templates are usually the first thing that every major release bork on, and all new model architectures end up having a ~2 week initial window of small fixes before they’re not DoA
z4y5f3 2 days ago [-]
Yes, and this is not the first time they messed up. They had tokenizer bugs where the trained weights do not match the template back to Qwen3 series.
Der_Einzige 3 days ago [-]
Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
MrDrMcCoy 3 days ago [-]
For those us us who don't know, what do those parameters do and why are they better?
suprjami 3 days ago [-]
Temperature, top-up, top-k, min-p all control which token the model predicts next and how likely it is to select one token over the other.
You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.
That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.
MrDrMcCoy 3 days ago [-]
I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.
suprjami 2 days ago [-]
Top-K: example setting 20. Select only from the 20 most likely tokens.
Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.
Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.
The purpose of all of these is to exclude very unlikely next tokens.
Der_Einzige 2 days ago [-]
Min-p is specifically "Don't select tokens less probable than a multiple of the top token's probability" with min_p of 0.1 multiplied by an example top probability of 0.3 being 0.03 as the truncation at that time step.
I've run the inference to get the answers I linked to. If someone else does the same thing, that involves extra energy. If I read your conversation instead of generating my own, then that's one less tree that has to be chopped down.
Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.
3 days ago [-]
3 days ago [-]
jmiskovic 1 days ago [-]
The Qwen team published the same sampler settings for 3.8 and presumably they used those while testing on benchmark. Do you believe they could have achieved higher result with top-n-sigma?
nullc 3 days ago [-]
Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.
Der_Einzige 2 days ago [-]
Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...
WithinReason 2 days ago [-]
Show us, we're curious. Did you upload to ArXiv yet?
Thanks this bumped my agent success rate from 67% to 92.5% (!!!)
zenoprax 3 days ago [-]
Depends on your tooling and quant? I grabbed the unsloth Q3 and it works out if the box in opencode. I had issues with OpenWebUI with a random 3.6 A3B.
Casteil 3 days ago [-]
One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
weitendorf 3 days ago [-]
IME this is a strong/reliable model smell, you typically see smarter and less benchmaxxed models' thinking traces spending more time exploring the solution space, and benchmaxxed models more time trying to refine/decide on the response contents. It has always been a big problem with qwen.
In general it's kind of a benchmaxxing/distillation (I don't mean that pejoratively here, read on) artifact where test time compute's purpose is to refine/zero-in on a particular input:output pair. Basically they're trying to "remember"/re-derive the answer rather than arrive at it deductively - it gets comical/absurd when you see it expending a huge amount of tokens trying to figure out the answer to some trivial question or response to some input, as if it were a trick question or the choice of wording was of the utmost importance. But it's not a bad thing when the output matters or the task is hard: basically the model has been trained to respond/act like a much smarter model and it's probably better for it to overgeneralize that behavior.
Also, IME Gemma and most other local models that support thinking tend to have the same problem, and it's partly only a "problem" because they give you access to the thinking tokens themselves (which you don't in many cases see when working with frontier american models) and you can actually see how they're getting spent. For the local or bulk (runnin on owned or rented hardware) use case where you are paying in time/watts rather than purely by the token, IMO it's a good idea to just look at the task success rate / time per task and not worry about the raw thinking token count.
Given that it apparently defaults to 'xhigh', this is probably the answer.
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.
eek2121 3 days ago [-]
I haven't tried lowering thinking, however, I actually asked a solid question earlier regarding a real world scenario I encountered and all that excessive thinking made it give me an amazing answer. The thinking actually all made sense, and honestly I found it thought of similar stuff to what I thought when I drew my own conclusion.
I don't usually rely on AI for much (I'm actually kind of anti-AI, although I follow stuff like this enthusiastically because of the rapid advancements, "average joe" access, and openness), however for what I asked? It was spot on.
The subject was a bit personal, so I won't share. I was just curious what AI would say about the situation and it definitely surprised me, especially since Qwen, while usually great on development/coding stuff, has shown weaknesses in other areas.
Definitely a solid release, and this one runs on my 4090 with minimal loss of quality!
me_bx 3 days ago [-]
In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.
Did I miss something, is it possible to have that model be reliable without thinking?
hadlock 3 days ago [-]
Turning off thinking mode you need a different tokenizer and a whole host of settings. It's more like switching your vehicle from gas to diesel, than turning your headlights from lo to hi.
dannyw 3 days ago [-]
The official docs recommend different sampler options for thinking off. Have you tried adjusting those?
me_bx 2 days ago [-]
Thank you and sibling poster for the suggestions.
I used the settings recommended on huggingface/unsloth's model page + whatever suggestion from various LLMs - didn't research too much myself which setting did what.
Der_Einzige 2 days ago [-]
Frontier model labs don't know anything about good sampling settings, and why should they? Their belief is that "sampling doesn't matter, we can simply scale our model and improve logprobs quality that way"
Ignore whatever nonsense qwen says about what settings to use. Turn off top-p and top-k. Turn on top-n-sigma and keep it at its defaults.
See repetitions that you don't like? Ignore the "repetition, frequency, presence" penalties and turn on DRY (don't repeat yourself).
Want more creativity? Temperature is okay but XTC (eXclude Top Choices) is better.
Llama.cpp quietly has had far advanced samplers compared to the shit stuff Qwen et al recommends for years.
IronWolve 3 days ago [-]
Thank you, this is exactly what I needed.
lrvick 3 days ago [-]
Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.
NamlchakKhandro 3 days ago [-]
really? crush.... it's trash harness compared to pi.
I didn't realise there are people out there unironically using crush
lrvick 3 days ago [-]
Yeah. I use it to do extensive work on full source bootstrapping, deterministic operating systems, compiler debugging, kernel debugging... all with one tiny go binary without endless NPM deps like pi (which /I/ regard as trash)
Works better than opencode (pi based) or anything else I have tried for my needs, and by far the prettiest and easiest to reason about what is going on.
But I will bite. What does pi do today better than crush for your use cases?
Rayosay 2 days ago [-]
What's wrong with it? I like Crush. It's hard to find good harnesses that don't pull in mounds of Javascript like Pi and OpenCode.
Frost1x 3 days ago [-]
“Thinking” is just a guiding methodology to help iterations (between the initial prompt, results, and a mixture of harness back and forth to the LLM) converge on something sane in a massive parameter space.
I like to think of it much like (as a common example most people can relate to) the Newton-Rhapson method for finding roots of a (mathematic) function. Your initial prompt runs, then the ‘harness’ kicks in using whatever methodologies are behind them to iterate on that prompt (back and forth with the model, occasionally with the user to get better guidance) and refine the outputs to hopefully converge back to some sensible output or actions the user was initially looking for.
So you’re hoping for an LLM that sort of ‘zero shots’ or needs minimal iterations from a prompt to give usable results. I find from my anecdata it varies across models and what I’m trying to get it to converge on. I tend to prefer models to not zero shot attempt because they tend to not do great, I want them to get feedback often to let me push them down the route of convergence in spaces I already understand well, meanwhile I like them to explore and give me new paths in spaces I’m not too familiar with.
That’s really what all that “second guessing” is, it’s making sure you’re following a sane path in a massive parameter space of an ambiguously defined problem. Imagine if in Newton’s method you checked the slope and it didn’t decrease from the last iteration and you just say “screw it let’s keep trying that direction.” LLMs and their harnesses tend not to have that base assumption like iteration on decreasing slopes to guide them closer to convergence, it’s a lot messier.
mrshu 3 days ago [-]
This is why the "reduce thinking" approach of BottleCapAI is both quite interesting and very useful in practice:
> As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time.
Not my experience at all. On my M1 Max MacBook Pro, Qwen 3.6 35B-A3B provides better responses in similar time (~60 tokens/second, vs. 50 for Gemma 4 26B-A3B). 3.8 27B is slower, being a dense model, but so are dense Gemmas.
cyanydeez 3 days ago [-]
You can add a thinking budget thats not much effort in llamacpp. You can align the cut off message with your agent instructions.
What you describe is a engineering harness problem.
If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck
This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.
Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.
Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.
ThouYS 3 days ago [-]
gemma4 can't hold a candle to 3.6
kimsey0 3 days ago [-]
If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.
wincy 3 days ago [-]
Amazingly enough, Hacker News decided to show this to me as the top comment, I'm also running an rtx 5090 and trying it out now, thanks for the tip!
Edit: Absolutely blazing fast! Getting 163 tokens/sec on WSL and it generated a pretty sweet Pelican.
I'm running it using a 4090 on using llama.cpp with Q5_K_S and its running at ~33 t/s
mirekrusin 2 days ago [-]
[dead]
glinkot 2 days ago [-]
Yep, same, testing it now and it flies!
onlyrealcuzzo 3 days ago [-]
If the benchmarks don't lie, this is getting very close to Opus 4.6 capability - which was the turning point for me for when AI was "good enough" that it became very hard to justify not using it.
I'm sure there's some benchmaxxing going on, and some things you get only with a a larger model.
But I'm feeling pretty confident if not by Gemma 5 than by mid 2028 we'll have local models that are almost always as good as Opus 4.6 was and in many cases far better.
DanielHB 3 days ago [-]
What kind of things you only get with a larger model?
redox99 3 days ago [-]
Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
anon373839 3 days ago [-]
> You just can't compress the entire human knowledge into a 30GB file
Fortunately, that isn’t necessary! What LLMs need is a level of fluency with key concepts so that they can (1) make effective use of retrieval tools and (2) understand the material in the context window. 30GB-sized models can absolutely store enough knowledge to do this.
Here is an example from the field of law. Most lawyers who have litigated contract disputes in California know about Civil Code § 1717, which makes any contract providing for attorney fees to a prevailing party mutual, so that even if the contract was written to be one-sided, it won’t be enforced that way. It’s a simple enough concept, but there are many more particulars to it, such as what happens when the fee provision is only written to apply to part of the contract. (Answer: it depends on other facts.)
When a lawyer recognizes that they’re in a situation where § 1717 is relevant, the first thing they will do is pull the statute and read it, because nobody has it memorized. And they don’t need to.
nullc 3 days ago [-]
> (1) make effective use of retrieval tools and (
A downside is that you can't just download a lot of that knowledge, vs with the weights the copyright infringement has been outsourced to the lab. Nor can you just search for the info because the internet as a whole is increasingly aggressive at blocking anything that looks like an AI agent.
I'd love to see more retrieval powered local AI-- I think it's an area that open source development could excel. ... but there are advantages of having the knowledge in the weights!
Perhaps what needs happen is for someone to make an "ultrapedia", an AI restatement of a huge library of reference works-- created expressly for the purpose of being a locally stored corpus for AI agents.
DanielHB 2 days ago [-]
I wonder how much more effective LLMs would be at general knowledge if you just download wikipedia and set up an MCP for it.
Wowfunhappy 3 days ago [-]
> You just can't compress the entire human knowledge into a 30GB file.
...I'm just asking questions here... how sure are we of this?
If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.
If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.
sbierwagen 3 days ago [-]
Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic extensively uses synthetic text in training. You could probably get good 50tile, 90tile, 99 etc numbers just from the size of the training materials on Anthropic servers.
On a more serious note, it depends on your cutoff for "entire human knowledge". It's easy to prove for a generous interpretations of "entire human knowledge" that it can't be done, but hard for something like "all useful human knowledge".
throwaway27448 3 days ago [-]
Wikipedia (text) is about 25gigs compressed. I think that's a reasonable starting point.
gen220 3 days ago [-]
IDK, 30GB is a lot of data when we're talking about text!
Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB.
I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class <insert name of preferred professional knowledge worker> in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).
redox99 3 days ago [-]
I think a specialized model could squeeze all you need to know about a certain profession in 30GB. But not all professions at once, which is what these models try to do.
Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.
gen220 3 days ago [-]
Yea to be clear I think >70% of the information is not profession specific.
I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.
The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.
My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.
redox99 3 days ago [-]
The human genome is 800MB, (and 500MB with compression according to GPT) so that part is easy.
I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".
Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.
Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.
anon373839 2 days ago [-]
> A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for
What makes it hard to search for? If you tell me you want to open a lemonade stand in East Bumbling Falls, I won't know that city’s municipal ordinances. But I will know that cities tend to have ordinances. So I’d search broadly for what codes that city has. Then I’d go through each one looking for anything that might affect a lemonade stand.
redox99 2 days ago [-]
Usually at least here it's tens of thousands of laws like "replace article 3 paragraph 2 with:" and it's not like you can ctrl f "lemonade stand" and get something useful.
A strong llm without that knowledge can probably spend millions of tokens parsing every single ordinance and give you an answer if you directly ask about that.
But both a local human or an "all knowing" llm when you just tell it about the stand will be like "hmm wasn't there a law about low calorie beverages being tax exempt? Let me check" and after a very precise search come back with a tip that if you sell sugar free lemonade that's tax exempt.
I mean would you get an accountant from another country just because he can use search?
DanielHB 2 days ago [-]
English Wikipedia is around 25GB compressed
yogthos 2 days ago [-]
[dead]
versteegen 3 days ago [-]
IME using 5.6 Luna and DS V4 Flash, I notice that although they are excellent at programming, even Opus-like in the way they try to debug, the thing they are worst at is inferring user intent and making good decisions with little information. They are absolutely terrible at that, will misinterpret small wording ambiguities. I suspect that's an ability you can't add with RL training, that it requires the depth of understanding from vast pre-training.
johnnyApplePRNG 3 days ago [-]
So add a pre-ingestion agent to your Pi Coding Agent subagent repertoire, problem solved.
solarengineer 3 days ago [-]
Please, could you point me to any documentation about this pre-ingestion?
The point is to simply add a subagent that interprets your task and hands that off to the coder set to the appropriate level of thinking, essentially.
NamlchakKhandro 3 days ago [-]
Woah woah woah buddy... suggesting that people use anything other than ClaudeCode or Codex is simply not allowed around these parts.
solarengineer 3 days ago [-]
Your negative impression is surprising. Could you point to any threads to support your message?
onlyrealcuzzo 3 days ago [-]
Similar to the way they asked Sol to solve Erdos problems, that's what I want my model to do for programming.
I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).
What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.
It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.
Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.
You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.
ferrouswheel 3 days ago [-]
You're going to be waiting for a while.
Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes.
Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.
johnnyApplePRNG 3 days ago [-]
I honestly think we've got about a year tops.
voiceeh 3 days ago [-]
I'd consider a year a while, but then, time is relative.
fragmede 3 days ago [-]
I've heard that some people even have a plan on how they'd like their life to look like in 5 years that they're aiming towards.
reilly3000 3 days ago [-]
I think someone ought to encode this into a harness. It is really insightful into how we should be spending time if it is going to be spent reviewing AI code.
alex7o 3 days ago [-]
This is a harness problem not a model problem, try prime agent it can do that and it will do it well even :P but you need to prompt it in according to its tools and processes.
aka-rider 2 days ago [-]
Smaller models are overconfident and have a hard time to self-correct.
If it’s stuck, usually that’s it.
Bigger models “understand” better, both the prompt and the contents. If you will try to read a paper together with a smaller model, the difference is immediately obvious.
Bigger models will “forget” and drift much less.
RandyOrion 3 days ago [-]
Thank you Qwen team for this release.
Compared to closed weight (especially unreleased and access-limited) and open weight/source large sparse MoE LLMs/VLMs, open weight/source small dense models benefits public the most because they just reaches more people.
Compared to Qwen 3.6, 3.8's thinking style changed drastically. With xhigh budget, it thinks a lot MORE, and longer thinking session directly translates to better performance. This tradeoff between performance and computation, memory, etc. is meaningful to me.
However, because Qwen 3.6 and 3.8 share the same architecture, with 32GB vram, llama.cpp, IQ4_XS model, MTP and FP16 mmproj, I can only get 200k context, which is not good compared to 640k context of muse glimmer. Hopefully this problem will be solved in Qwen 4.0 release.
Overall I'm very impressed with how well this did. It's a big improvement over 3.6, and it feels on-par with some much, much larger models. I think this one is on-par with Gemini 3.7 Flash.
One thing to note - the build for this on my RTX 6000 pro blackwell took a long time. Easily one of the longest builds I've done. It took around 2 hours to build the site. Obviously we'll have some quants for this soon that will accelerate things, but I was still surprised with how long it took.
harness setup? how much vram ? how are you handling a 2hr long build ? multiple sessions? fan out sessions (subagents)?
tomr75 3 days ago [-]
what token/s?
jjcm 3 days ago [-]
27 t/s. I suspect there will be significant speed ups in the coming weeks.
apitman 3 days ago [-]
You should be getting way more than that on a 6000 pro even today. I'm getting 40tok/s on a pair of 3060s. You can ask a SOTA model to optimize your setup for you.
dexterlagan 2 days ago [-]
Getting 30 t/s on a Mac M5 Max laptop. You should be getting close to 200 if you can use the 5090 acceleration tools I see posted here. There are forks for the 4090 and the 3090. Maybe there's a fork for the 6000?
graceful6800 2 days ago [-]
I'm seeing the exact same number on my Blackwell box. MTP put it at almost 70 which is pretty decent
nojs 2 days ago [-]
Any idea why it’s so slow? the entire model should fit in the vram of one card.
graceful6800 2 days ago [-]
The NVFP4 quant is completely broken, so I'm not shocked that other quants aren't fully there yet. Give it some time to cook.
svdr 3 days ago [-]
Wow. This model is so good, and we have GLM 5.3 (seems great voor security related work) and Deepseek. In a few months we'll have Fable/Sol-like capabilities that are not coming from the big US companies. I feel as a programmer that that is more than enough. How wil OpenAI and Anthropic survive when frontier model intelligence becomes commoditized?
chr15m 3 days ago [-]
I stopped using Fable and Opus 5 because I literally can't understand the output. The waffle is so intense it no longer makes sense. I don't understand who is using those to get real work done.
I'm using kimi-k3 on a real high level dev & analysis task at the moment (with Gemini flash for fast implementation) and it is fantastic.
I think we have passed the point where frontier intelligence is commoditized.
just need to give it clear direction, guide it about how to properly use hooks / preserve decomp code, rewrite all the comments, go back & redo things when it spits out some ugly hack, ...
but this does save time, especially considering this is working with decomp code where many functions are random hex addresses
gen220 3 days ago [-]
just curious; which fable/opus versions are you using?
Fable-1m-max-thinking output, despite being incredibly slow and expensive, feels like it bucked a trend towards superficial loquaciousness in their models that had been building since 4.6.
I'm a bit of a luddite when it comes to upgrading models, fable was the first one to make me give up Opus 4.6-1M-max.
chr15m 3 days ago [-]
I'm just using the defaults, through Claude.ai on the web.
pmarsh 3 days ago [-]
Curious what level of thinking you have been using as well. Found that lowering it actually gets me better results.
chr15m 3 days ago [-]
I'm just using the defaults.
horacemorace 3 days ago [-]
They’ll certainly try to stymie people by colluding with manufacturers until we get nvidia level hardware or LLM ASICs from the East.
awb 3 days ago [-]
The demand curve for speed and intelligence seems pretty steep to me.
If you look at the hiring marketplace, being just marginally better than your peers can be very lucrative.
If you’re competing on speed or capability as a company (or as an employee), you’re probably going to be willing to pay for the frontier.
HarHarVeryFunny 2 days ago [-]
> If you look at the hiring marketplace, being just marginally better than your peers can be very lucrative.
I would say that in software this is completely false.
Someone straight out of college, not very useful, makes 75-100K.
Top level senior outside of FAANG is making twice that at best (and at least 10x more capable).
Der_Einzige 2 days ago [-]
You told on yourself about being either European or from a flyover state.
HarHarVeryFunny 2 days ago [-]
I happen to be from NJ, but you're also not going to be making a Meta/Google salary in NY outside of NYC.
croes 3 days ago [-]
Most companies have a limited budget.
Good enough with a lower price will win the masses
KronisLV 3 days ago [-]
I hope really badly that we'll get a new 35B A3B or similar MoE model!
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
I'm hoping too that they'll put out some MoE variants.
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful.
I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
Yeah, that's probably the answer given that it apparently defaults to 'xhigh'.
dannyw 3 days ago [-]
Probably helps it score a little bit better in benchmarks :) `medium` seems like a nice balance so far; along with some light steering to vary think effort as needed for task and being pragmatic.
wgd 3 days ago [-]
Interestingly 'medium' is the closest thing the _model itself_ has to a default thinking level. The chat template injects directions [1] at the very start of the system message when the reasoning effort is 'xhigh' or 'low' but 'medium' implicitly just means no added reasoning-level instructions.
[1] The specific directions are "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer." and "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
mirekrusin 2 days ago [-]
[flagged]
isomorphic 3 days ago [-]
"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour!
The MoE models are stupendously faster.
kzrdude 3 days ago [-]
I think that's called test-time scaling i.e using more tokens at infer time to squeeze out higher model performance. That's must be part of the explanation for good benchmark results.
try-working 3 days ago [-]
that seems to be how most Chinese models achieve increased benchmark scores. GLM and Kimi models are "thinkslop" models that reason over their own thinking, which increases cost and decreases speed significantly. That's why GPT is in a different tier altogether - faster, smarter, and sometimes cheaper.
Casteil 3 days ago [-]
Yep.. it's pretty obnoxious for real-world use with the default 'xhigh' thinking. Ridiculous amount of "Wait, actually.." which might help for complex coding tasks but makes it unbearable for general purpose use.
nubg 3 days ago [-]
what harness did you use?
isomorphic 3 days ago [-]
No agentic harness; I just prompted literally "svg owl" into LM Studio 0.4.21+2. macOS Tahoe 26.6.1, Mac mini M4 Pro (14 core, 20 graphics cores) 64GB RAM. (I put the machine config so the 17m12s of thinking has some context.) It got about 12.75 tok/s and used 21,769 tokens.
For comparison's sake, Qwen 3.6 35B A3B MLX, same prompt, same machine, thought for 1.59 seconds, used 2,398 tokens, and hit 80.83 tok/s. It actually output just a straight SVG file of an owl (not a bad owl either!), whereas Qwen 3.8 27B spent a zillion times longer gold-plating everything and output HTML+JS+SVG with interactive animation, sound, etc., etc.
Phemist 3 days ago [-]
Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.
peri-cl 3 days ago [-]
[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
Tested:
Qwen3.6-35B-A3B-UD-Q4_K_XL
Qwen3.8-27B-UD-Q4_K_XL
qeternity 3 days ago [-]
35A3 might be more comparable to 10 dense.
27 dense is far more capable than 35A3.
kzrdude 3 days ago [-]
So you happen to know how to demonstrate this higher capacity, what do we mean by that? Can understand bigger codebases? And how do we explain it, what part of the model does it?
searealist 3 days ago [-]
If you are not already using MTP, you should be able to get ~2x decode tokens/s with Qwen 3.8 27B.
cyanydeez 3 days ago [-]
yeah, that's the A3B part; going up to A5B would probably also feel comfortable.
on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.
rasengan0 3 days ago [-]
Got Qwen3.8 to run on my Framework 12 Intel Core 13 Gen Raptor Lake i5-1334U small laptop with 48G RAM stick:
but it failed my basic prompt to compose a vim regex to match CamelCaseWords
downgraded a bit with Q4_K_M from ollama run qwen3.8:27b
and /set nothink and at least 1 regex matched FooBar
prompt eval at 2.8 t/s
eval at 0.94t/s
I particularly enjoyed this usage of the regex:
/%\1\%/ ... onward for 700+ characters of \%\/
:-)
expedited123 3 days ago [-]
Mind sharing your laptops specs? Just interested to see what is needed to locally run Qwen3.6-35B-A3B
peri-cl 3 days ago [-]
Don't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4))
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
expedited123 3 days ago [-]
Thanks! I don't have any knowledge of running models locally.
I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
seanmcdirmid 3 days ago [-]
not parent, but 4-bit quantization is generally consider a good trade off for speed/performance, so you might use it even when you aren't on consumer hardware, but definitely when you are on consumer hardware.
SwellJoe 3 days ago [-]
AgentWorld is a pretty recent MoE release from the Qwen folks with quite a bit better performance than 3.6, released around the same time as 3.7 Plus/Max, I guess. Not sure why it didn't get more attention, as it is definitely better than 3.6 35B A3B on all dimensions, but especially for agentic use. Still nowhere near the dense models (even 3.6 27B), but clearly an upgrade in the small Qwen MoE line.
KAT Coder is another Qwen 3.6 MoE fine-tune that also improves on 3.6 by a measurable amount.
But, I do hope for a bigger MoE, in the 70B to 120B range, something in the Coder Next lineage. I've got a Strix Halo that isn't getting used to its best ability because the best models all run fine on my faster desktop dual 32GB GPU setup, and they run too slow for comfort on the Strix Halo.
nullc 2 days ago [-]
HUH? AgentWorld is a simulation of the world (e.g. tools and programs) for use by an agent!
SwellJoe 2 days ago [-]
And, yet it outperforms Qwen 3.6 35B A3B on Terminal Bench and SWE, etc. I dunno.
Edit: I guess you're right; apparently it's for simulation. I didn't look into it beyond the benchmarks. But, it does work in an agentic context, regardless. It'll write code, and drive an agent.
jwr 3 days ago [-]
Me too. 35B A3B runs really fast on my MacBook Pro (M4 Max) and is suitable for real-time tasks like dictation post-processing. The dense model is not.
colordrops 3 days ago [-]
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
pickledish 3 days ago [-]
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)
Fervicus 3 days ago [-]
It's better for VRAM poor people. I get 4-5 t/s with 27B and 20-30 t/s with 35B A3B.
dofm 3 days ago [-]
Also radically better on an M1 Max. I get well up into the 60s t/s with the A3B, stuck at 9.5 or so with this new 27B, though perhaps an MLX build will help.
cmrdporcupine 3 days ago [-]
MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.
Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.
nullc 2 days ago [-]
35Ba3b is usable in plain cpu inference on a fast server, the dense model is MUCH slower. On GPU the 35ba3b is still around 2x the tok/s single threaded, which can be a good tradeoff for some applications.
jakswa 3 days ago [-]
anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.
formerly_proven 3 days ago [-]
Runs faster on bad hardware.
apothegm 2 days ago [-]
Or even on fairly high end (by general public standards, not gaming community standards) consumer hardware. Fantastic fit for M-series Macs with 32-96GB RAM.
Alifatisk 3 days ago [-]
> I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
kennywinker 3 days ago [-]
Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s
Alifatisk 3 days ago [-]
Thank you Kenny
martinald 3 days ago [-]
You can run these on CPUs at a somewhat reasonable speed.
KronisLV 3 days ago [-]
Or (somewhat) low TDP GPUs for that matter, like workstation ones, that might have enough total VRAM but not the best bandwidth/compute.
ramon156 3 days ago [-]
People will claim it's not comparable to Opus despite it beating the score. I'm not sure I disagree, but I'm also unsure whether I care. Most new models nowadays are "good enough". I cannot complain because I'd rather spend that time improving my prompts and docs. Opus might be a _slight bit better_ at picking up vague hints, but it's also extremely expensive, and I hit the 5 hour limit way too quick.
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
simplyluke 3 days ago [-]
I'm convinced a lot of the anti-open-weight model comments at this point are inorganic traffic - there's trillions in investor money riding on a world where these models aren't cheap commodities. Having actually used things like the recent GLM, Kimi, and Qwen I think any edge the labs have is marginal at most and actually prefer the open weight models in most day to day usage.
Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.
anana_ 3 days ago [-]
When stuff like this: https://doublespeed.ai/
exists I don't find that hard to believe at all, although it cuts both ways
qlte 3 days ago [-]
Backed by a16z... yep, that tracks
phatfish 3 days ago [-]
Amazing, looking forward to stuff like this killing social media. Or at least forcing the use of robust ID/KYC checks to verify human accounts.
orangedog 3 days ago [-]
Why are you convinced of that? Pretty much every time I see statements like that online, I can find plenty of organic traffic supporting it, not everything is a bot.
I just wouldn't bias myself that way, most people haven't really used local models. This stuff is pretty much all subjective evaluation, there's plenty of reasons for people to favor certain models or disfavor others.
delduca 3 days ago [-]
Opus is pretty dumb and lazy nowadays, cancelled my subscription.
hypfer 3 days ago [-]
> I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting
Dude, GLM-5.3 released _today_.
The phrasing "I've settled on" is incorrect for this context.
stusmall 3 days ago [-]
I'm generally extremely skeptical about a lot of the model hype that show up in comments. Except when there is an extreme mismatch the performance, quirks and quality of these things are difficult to nail down. You wouldn't know that from the comment section of every single release.
I think some of these are excited, eager users always ready to hype up the new thing. The same crowd that previously would constantly push for a rewrite from angular->react->svelt->god knows what. Instead now it is on a 6 week cycle and about models/harnesses.
I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
Ultimately, I think some of it is legitimate shifts in who's in lead and what is the best. You gotta dig through a lot of crap to get to that, and I don't really know how to to.
Ultimately I'm saying is that I always applied a fair amount of skepticism about what I see in comment sections but these day it is extreme amounts.
EB66 3 days ago [-]
> I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
I've assumed the same as well.
I also assume that many of the companies developing these models engage in benchmaxxing.
At my company we've developed our own internal benchmarks for evaluating LLM models as they become available. The benchmarks are tailored to our particular use cases but the utility and knowledge our benchmarks assess is still fairly universally applicable. I see wide differences between what our internal benchmarks report and what the major benchmarks do.
There was a whole lot of fanfare about how amazing GLM 5.2 was when it was released, but it was pure rubbish on our internal benchmark -- far behind OpenAI, Anthropic, Gemini, DeepSeek, etc. I don't know how to reconcile the fact that GLM 5.2 performed very well on some of the major public benchmarks, but consistently performs so poorly on ours. OpenAI models tend to dominate our internal benchmarks.
gpt5 3 days ago [-]
It actually is showing in public benchmark if you know how to look for it. For example, in Terminal-Bench 2.1, GLM 5.2 received 78%, while GPT 5.6 Sol received 88%.
Then Terminal-Bench 3.0 came out (where the questions are new), and GPT 5.6 Sol received 34.6%, while GLM 5.2 dropped to a whopping 4.6%.
gr_norm 3 days ago [-]
> OpenAI models tend to dominate our internal benchmarks.
That's odd, since Fable seems to be the leader for the industry. Not cost-effective, but if Anthropic models get dominated by OpenAI in your internal benchmarks, this calls their validity into question. Separately, see the jagged frontier effect. [1]
I've been using GLM 5.2 at my day job (mostly Rust backend work ATM). Nothing that blows away the models from OpenAI and Anthropic, but solidly good enough to get it done. A lot of people have experienced this and the fact that an open weights model can do so is where most of the excitement comes from. Optimizing for benchmarks can only get you so far, and people are quick to criticize models that fall into it (like DeepSeek Pro V4 recently).
This resonates. You need to tailor evaluation to your own use cases in order to know if you should adopt the next thing or not.
bliteben 3 days ago [-]
I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
Really no different than when there was suddenly online personas everywhere hyping up TSLA out of the blue. You can see the same thing going on with the BoringCompany subreddit. Crazy that the botnet master isn't able to convince us that Grok is also the best model. I don't think buying twitter was an accident it was probably just literally covering up the evidence.
IMO this is getting hyped because the 27b version runs on a decent gaming GPU. This is NOT a thread for their largest model, this model will run on a mid-high end gaming PC, which you probably have in your household. Mine is 6 years old and it runs quite well.
dofm 3 days ago [-]
I strongly suspect that many of these accounts you think might be bots from the labs are just people who only have one interest.
One thing I have noted a lot more of is that there is comparative fanboying going on. Like "this model has done badly, my favoured competition has a model out soon that will beat this in every way" — comparing a released product to unverifiable hopey claims about an unreleased product.
It's tempting to assume that is bot stuff, but if you've been around any other "hot" technical hobby online (cameras, phones, 3d printers, whatever) you will know it's not. It's just fans aligning into teams, some of them laconic and amusing, some of them overkeen and toxic.
ramon156 3 days ago [-]
hence the "former deepseek v4 pro". I tried it out this morning and have had no complaints. I already liked glm 5.2
kristjansson 3 days ago [-]
> Deepseek v4 pro 0813
Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?
throw310822 3 days ago [-]
It's the singularity, baby.
mlsu 3 days ago [-]
He's 100x, bro. 36 hours is basically like a whole 2 week sprint for you.
hypfer 3 days ago [-]
The sentence still doesn't make sense, because "settled on" implies a long testing phase with a verdict eventually emerging out of that.
What you're currently doing is "testing out"
Topfi 3 days ago [-]
Honest question, how do you assess models this quickly? What metrics are you using? Would love to get my suite from multiple days and hundreds of prompts down to minutes. Got a few first pass tasks I run upon release for an initial experience, but those only work because even Fable and Sol fail despite objectively correct solutions existing, so it works because most models fail, but then, those are consciously not enough for coding, tool use, adherence or task specific inference and assessment…
satvikpendem 3 days ago [-]
What are you working on? That can dictate which models are best.
Topfi 3 days ago [-]
Anything from ML pipelines for language specific pruning over a Rust/JS/CSS mix codebase to assistance in motorcycle maintenance and different canvas coatings. Most of my evals build on those requirements and especially past failures, whether in pure information, task execution and coding or tool calling beyond the overfitted mainstream. All stuff derived from actual failures encountered, some still only few models come even close to passing. With such a mix, it just takes a while to get any serious opinion on a model. Doubt anyone can do that in such short time, unless their tasks are so simple that most modern models not only succeed but could themselves accurately rate output. If even Fable still confuses PU or wax coated cotton canvas with a nylon shell, or tells me with a straight face to adjust the valves on a bike that has hydraulic lifters that needs experience for human assessment and the time that comes with it. Anyone with less knowledge either wouldn’t see the mistakes staring them in the face and just go by vibes, any model rating these equally can not tell what is accurate and will just go by the output sounding accurate over being. Gives sometimes very interesting results far different to public benchmarks. Inkling, e.g. is more accurate in not telling you to adjust valves that are simply not adjustable then Fable or Sol, which just tell you to adjust every 5000km. Sometimes even when their reasoning and search includes sections about the fact this is not necessary or possible. The beauty of overfitting and unbalanced training data…
arjie 3 days ago [-]
Most of us aren’t deploying these in general purpose use-cases. E.g. I use Qwen mostly for vision in my personal assistant. I have an eval set for that. Pretty much each of my use cases has a pre-computed problem set.
Likewise I have a tool-use eval set and a browser-use eval set. I use frontier models for most interactive tasks that are not home assistant or background agents.
I can imagine someone could build evals for that but I have never done so.
Topfi 2 days ago [-]
Deepseek v4 Pro or GLM 5.3 for software architecture I feel are only deployed for general, rather doubtful those are for narrow stuff, if we are sticking with this threads mentioned models. For small models, sure, narrowly targeted sets which can be self evaluated are amazingly valuable, but I feel beyond 500B we are in a different dimension. Rating any model the size of GLM or V4 Pro in hours I doubt is done beyond pure vibes.
lacoolj 3 days ago [-]
Grok 4.6 is a game-changer. I have yet to go back to other models after starting to use it. You just can't beat the price + output quality (even K3 is more expensive)
pbhjpbhj 3 days ago [-]
Supporting a far-right megalomaniac, whilst helping them to train their ML, and giving them all your data ... what could go wrong.
whatshisface 3 days ago [-]
I try to do everything with open-weight models, to support the companies who support me.
d2kx 3 days ago [-]
> Alibaba Cloud subscriptions are hard to come by
no they aren't. they discontinued their always-sold-out coding plan and launched QwenCloud (basically a friendly frontend with Alibaba Cloud as the hidden backend) and launched typical subscription plans for Qwen & co alongside it.
tosh 3 days ago [-]
i think you will like luna if you haven't tried it yet
johnnyApplePRNG 3 days ago [-]
Luna is twice the price of Deepseek V4 Flash 0731, and less capable :/
declanjackson 3 days ago [-]
With max reasoning, Luna is actually less than half the actual cost to run compared to DeepSeek V4 Flash 0731 with updated prices (based on Artificial Analysis Cost per Task)
wyre 3 days ago [-]
How is it less capable? I get being cheaper, but I’m pretty sure Luna is the stronger model?
The price on DeepInfra and most providers on OpenRouter has not changed... if anything, it's gotten cheaper.
All of this DeepSeek price increase stuff is related to their specific API, not the entire market.
farlight 3 days ago [-]
They were significantly more expensive for coding compared to DeepSeek's offering because of much higher cost of cache reads and lower cache hit rate. Now, after the price jump, the effective cost of deepseek.com API vs 3-rd party providers is about the same.
johnnyApplePRNG 3 days ago [-]
DeepSeek's API was never "cheap".
They were stealing your code for that extremely cheap cached token price.
That was never the real price. Every Chinese LLM API provider is the same to the best of my knowledge.
"CHEAP API" (but pssst we train on every single input aka your code and ideas are ours forever)
hirako2000 3 days ago [-]
There is no more, no less, guarantee that U.S providers don't do exactly that. In fact openAi is going to court.
If some models are cheaper it's simply because they cost far less to develop. And/or run.
127 2 days ago [-]
And you're claiming Anthropic and OpenAI are not doing the same thing?
stavros 3 days ago [-]
Looks like it's changed now, OpenRouter shows providers having the updated DS4 pricing.
alecsm 3 days ago [-]
Up to 4x but in 2 days.
dools 3 days ago [-]
No way is it less capable. When deepseek can't get its shit together, I give Luna a go, then Terra, then Sol.
Most of the time Luna figures it out where Deepseek was failing. I rarely have to go to Sol levels.
focxle 2 hours ago [-]
[flagged]
satvikpendem 3 days ago [-]
You should check out Grok, it's quite a good deal from the Cursor subscription side but it's cheap even by API prices.
gpm 3 days ago [-]
I think it's very clear that someone who has checked out all the models but the one that called itself mechahitler and is explicitly being fine tuned to support far-right politics is making the choice for reasons other than performance and cost. It's not like all the other models even had plausible claims to those metrics.
satvikpendem 3 days ago [-]
As I said to a dead reply, for coding all of that is immaterial, as long as it codes well then that's all that matters to most people, except it seems those who have an idelogical issue in which case the other model companies also have issues.
gpm 3 days ago [-]
No, coding does not make ethical issues immaterial. Nor does it justify your behavior of harassing someone to do something that they've clearly decided not to do for ethical reasons on the basis of criteria that they have clearly rejected.
satvikpendem 2 days ago [-]
"Harassing" lol by telling them to try a model, alright. Anyway I didn't know they didn't want to use it for ideological reasons, they didn't say anything about that. Now that I know there's nothing more to be said.
InsideOutSanta 3 days ago [-]
I don't care how good Grok is, it will be a cold day in hell when I give a guy who goes to AfD rallies, instrumentally supported the current Ebola outbreak, and gives the Hitler salute on stage (don't try to tell me he did not, I saw the video) any money I'm not absolutely forced to.
And yes, OpenAI and Anthropic are highly questionable, too, and so I've ended my OpenAI subscription and I'm moving from Claude to other providers. But even people who do not do that are absolutely justified in drawing a line at supporting Musk, because there's a difference between mere normal shittiess and Musk.
satvikpendem 2 days ago [-]
Great, don't use it then.
virgildotcodes 3 days ago [-]
The funny thing is it doesn’t even code well. It’s still worse than OpenAI, Anthropic, and the leading open source models.
So you get worse outcomes while also financially supporting one of the most prolific pushers of race wars, the destruction of labor rights, environmental protections, and general misinformation.
Truly a win-win?
Contrary to your framing it seems to only reason to use Grok at this point would be ideological.
satvikpendem 2 days ago [-]
It's a better deal compared to those is what I said. It codes well enough so that's why people use it. As I said, go to r/cursor and you'll find loads of people using it while OpenAI and Anthropic are more expensive per token and give less usage.
ai_fry_ur_brain 3 days ago [-]
[dead]
altruios 3 days ago [-]
[flagged]
satvikpendem 3 days ago [-]
Yeah I'm sure everyone on r/cursor or in previous HN threads about Grok 4.5 or 4.6 are all unserious and insane.
No one actually cares about the politics as long as the model codes well.
It's true that Cursor gives a lot of usage with Grok, most users of Cursor don't care about Musk.
altruios 3 days ago [-]
There are a lot of people who are apathetic to what musk is, people that don't care are not people who should inspire you. What the hell is so inspiring about apathy anyway?!
And yeah, people that don't care DO make the world worse through their apathy.
oezi 3 days ago [-]
I would say it is sad that there are people who use Grok when there are so many other choices available which don't come with the issues of supporting Musk.
It is not all just 'politics'. Take a stand on some issues. It doesn't cost much not to use Grok.
freshnode 3 days ago [-]
Voting with your wallet is still very much a valid way to protest that odious man.
Some people might not mind (or even know), but I sleep better at night trying to work as ethically as I can.
sixothree 3 days ago [-]
> No one actually cares about the politics as long as the model codes well.
I do. There are enough models to choose from that I don't need to use one from a guy who did a nazi salute at the presidential podium, and proceeded to rid our government of brown people he felt didn't deserve their job.
pauletienney 3 days ago [-]
"No one actually cares about the politics as long as the model codes well."
> I totally care about politics, especially when it comes to not giving my money to people like musk.
jLaForest 3 days ago [-]
i care about not financially supporting a person that is actively trying to disenfranchise me, why is that a difficult concept for some people? that not everyone is motivated exclusively by financial profit? is moral bankruptcy so pervasive that some people assume it is unanimous?
altruios 3 days ago [-]
'They' - in this case - are those who are unable to empathize. People who only learn of other's struggles through visceral personal experience. Conservatives, in general, struggle with this class reasoning... Very much a 'if I don't experience it then it doesn't exist' kind of mentality. Frustrating and exhausting. A lack of imagination, a lack of empathy, they just are lacking in these regards.
rootusrootus 3 days ago [-]
Your comments are all in absolutes. As if this was the one thing that mattered to you, everything else is irrelevant. Don't most people make their decisions based on a balance of considerations? Dismissing half the world as nazi or fascist does not seem productive, it kinda makes you look like the insane one.
kennywinker 3 days ago [-]
This is the “Mussolini made the trains run on time” of ai hot takes.
(Btw, mussolini didn’t make the trains run on time)
albedoa 3 days ago [-]
Also: "What do you mean everybody at this party is a Nazi? They seem uninterested in politics, and they have been so welcoming to me!"
tills13 3 days ago [-]
"Sure I'm indirectly funding the erosion of basic human rights in the States, but at least I made my Hello World app cheaper!"
cleaning 3 days ago [-]
[flagged]
jedbrooke 3 days ago [-]
I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page)
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
prometheus1992 3 days ago [-]
16GB mac mini - what chip? m4 pro i assume?
jedbrooke 3 days ago [-]
nope just my normal m2 mac mini. I bought it just as a normal computer to do my taxes and whatever, so it’s mind blowing that I can run this kind of AI workload on it. Well, “run” might be generous, it gets like 3-5tok/s
I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much
prometheus1992 2 days ago [-]
ahh gotcha. that speed sounds about right. i am running qwen 3.6 35bA3B(group-32 int4) on my m3 air (16gb) and get about 8 tokens per sec. i have not yet tried the 3.6 27B.
spwa4 3 days ago [-]
Sounds like you need to check what the max context is set to ...
jedbrooke 3 days ago [-]
100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much
dofm 2 days ago [-]
So far what I am seeing in my seemingly simple "Wordpress last-login plugin" test is that in xhigh reasoning mode (the default, seemingly) it overthinks so badly that it writes terrible bushy code with edge cases caused by going down rabbit holes.
In "medium" reasoning mode, you get the classic Qwen wait/actually thinking loops you see in 3.6 that I guess will need to be interrupted in the way others do already with an over-thinking guard proxy.
(In one of my test runs it is now on "OK TRULY FINAL APPROACH" after having got through "FINAL FINAL APPROACH". Can relate)
It gets stuck in a thinking loop regarding the WordPress API and (resolvable) ambiguity in my prompt, that I guess might be resolvable with a custom skill with hints on how to look it up (and maybe with the devdocs MCP).
In Low reasoning effort mode it flies through the task and writes pretty solid code. So maybe it is me overthinking what is needed here...
nubg 2 days ago [-]
how do gemma4 or muse perform?
dofm 2 days ago [-]
Gemma 4 26B does really well at this specific task (and a general MySQL-related puzzle I test on). I rather like it and now they have fixed tool calling, I would use it. I think maybe it has been trained well with "consumer" programming languages like PHP that are sort of commonplace things people want to do. I think for less commonplace programming languages, maybe it's worse.
Muse Glimmer thinks well and codes well in my tests; it does fine at this. I really like it so far, but my tests are fairly shallow.
One thing I have been struck by — my prompt includes this sentence:
"Please read the following and then ask me any further clarifying questions you need before proceeding with code generation."
Almost all models I've tested interpret this as an instruction to ask questions regardless. Qwen 3.8 27B is the only one that either expresses confidence that it doesn't need to ask clarifying questions, or in higher reasoning effort ultimately asks questions, but offers up defaults I can choose with a simple reply.
hypfer 3 days ago [-]
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
Aurornis 3 days ago [-]
> --cache-type-k q8_0 --cache-type-v q8_0
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
hypfer 3 days ago [-]
Yes to both.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is.
I think when tuning 3.6, I saw more like 33%? But not 100% sure.
hedgehog 3 days ago [-]
Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.
hypfer 3 days ago [-]
I wonder if we could take a page out of the Solar power book and do MPPT but for draft count.
Constantly sweep through possible values to find the best result for the current conditions.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.
ndriscoll 3 days ago [-]
Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
hedgehog 3 days ago [-]
This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
hypfer 3 days ago [-]
Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33
The unweighted average is 4.1
So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.
Also, cool that it carries over from qwen3.6 like that.
freehorse 3 days ago [-]
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
tyfon 3 days ago [-]
To add for me (and claude assisted but tuned myself), here is for 3060 12gb + 5950x 128 gb ram without image support: https://pastebin.com/HsAUBGSY
Thanks for this! I got the same card, but not the same CPU nor RAM, but this is a great starting point to start tweaking
acidburnNSA 3 days ago [-]
How many tokens/second you getting? I have the same CPU but a 5070 and only 64 GB of ram. I just got llama.cpp built and am now hitting a whopping 5-6 tokens/s.
jacquesm 3 days ago [-]
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
hypfer 3 days ago [-]
Problem with that is I think that it quickly devolves into cargo culting, nonsense and noise.
Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.
Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.
So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.
And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.
The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.
jacquesm 3 days ago [-]
I've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time.
Let me return the favor:
Deepseek V4 Flash 0731 Q8
This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.
I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.
jacquesm 2 days ago [-]
The next morning: if you have long common prefixes then you may want to increase the --cache-ram setting.
__mharrison__ 3 days ago [-]
As a MBP and DGX spark owner, I would love such a site... (Feel like it would be a low effort feature of hugging face).
Searching through Reddit and forums for best commands is annoying.
jacquesm 3 days ago [-]
Looking forward to having your problems :)
beembeem 3 days ago [-]
What stackoverflow should have become.
fragmede 3 days ago [-]
still could! there's the network site https://ai.stackexchange.com/ just need people to ask questions and to downrank accepted answers older than a year (that might not be possible though).
rs38 3 days ago [-]
pretty sure this exists already...
FridgeSeal 2 days ago [-]
Feel free to link it if it does…
secretluver 3 days ago [-]
[dead]
silentbob7 1 days ago [-]
Here's my model.ini for 4090 (using the chat template from froggeric):
version = 1
[*]
; Global defaults shared across all presets
threads = 16
[Qwen3.8-27B]
; Model files
model = ..\Qwen3.8-27B-FP8\Qwen3.8-27B-UD-Q4_K_XL.gguf
I can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]:
1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.
2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.
3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.
You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).
Congratulations, you're now running Qwen 3.8 27B.
Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
reilly3000 3 days ago [-]
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
hypfer 3 days ago [-]
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.
But maybe someone else has experience to share there
nubg 3 days ago [-]
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
bilekas 3 days ago [-]
You sound like your trying to reassure yourself of something.
I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
badsectoracula 3 days ago [-]
> I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
Your boss is a human.
This is a computer program running on your PC.
I hope you can see the difference.
3 days ago [-]
formerly_proven 3 days ago [-]
> I sure hope my boss doesn't think he built my work!
Most managers do though?
bilekas 3 days ago [-]
I've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine.
But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.
Who created it is still me.
fragmede 3 days ago [-]
Did Steve Jobs create the iPhone?
bilekas 3 days ago [-]
No, he did not. When it went wrong for v4 with the radio. I didn't see him take the blame.
reilly3000 3 days ago [-]
Would it exist in it’s present form if he didn’t exist?
bilekas 3 days ago [-]
this topic always goes into this level of ridiculousness. Your question can't even be answered. We should focus more on encouraging discussion, not asking unanswerable questions. It wont bring anything other than points of both sides with no objective correct answer.
3 days ago [-]
effdee 3 days ago [-]
Some people will now argue it was the chisel—not Michelangelo—who created David.
b112 3 days ago [-]
No, it's the difference between management and direct work.
None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.
sejje 3 days ago [-]
The idea guys will really think themselves gods, now.
effdee 3 days ago [-]
Well, this universe might be God's own AI slop.
smallmancontrov 3 days ago [-]
"Carve me a naked guy. Make no mistakes."
hypfer 3 days ago [-]
I cannot fulfill this request. I am programmed to follow safety guidelines that prohibit the generation of sexually explicit content.
williamcotton 3 days ago [-]
I generally agree and expect this to be the case from a legal perspective.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?
Absurd logic. The AI built the website.
mixologic 3 days ago [-]
Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
tinfoilhatter 3 days ago [-]
The director receives accreditation for directing the film, not creating it.
onraglanroad 3 days ago [-]
Doesn't the director generally receive more credit than the producer?
How many films do you remember the producer above the director?
tinfoilhatter 3 days ago [-]
What does it matter who receives more credit? The producer produces the film and the director directs it. If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it.
If there were multiple people involved in the creation of a film I helped to create, I cannot factually say I created it. Just like if someone builds something using code generated by AI, they can't factually say they created it.
swores 2 days ago [-]
There's a bit more nuance to it though, I think, because traditionally there has been more of a firm line between humans and tools.
For example, most people would agree with this line you wrote: "If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it." A pedant could say "woah hang on, that's ignoring the fact that actually the iPhone is the one recording the images that make up the film, how can the human get all the credit", yet nobody would actually make that argument when discussing who created the film.
Generative AI is pretty much the first time (maybe there are some niche contradictions to this claim?) we consider a tool to be contributing enough creativity to the process that we don't all agree "only one person was operating this tool, so that person is the sole creator" - but some people DO still hold that line, and do consider the human who wrote the prompt to be the creator.
And I don't think there's any objective technical metric we can use to say who's right, it comes down to our collective judgement deciding where the line is.
3 days ago [-]
dd8601fn 3 days ago [-]
That’s an interesting way to look at it, and still generous to the llm.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
It all feels a little like that, again.
sampullman 3 days ago [-]
I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.
tinfoilhatter 3 days ago [-]
I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.
sampullman 3 days ago [-]
I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.
rob 3 days ago [-]
I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.
tinfoilhatter 3 days ago [-]
I'm not sure that typing messages to a chat bot requires much catching up, but since you think this is what I should do rob, I better listen!
rapind 3 days ago [-]
> I'm not sure that typing messages to a chat bot requires much catching up
Lol, yeah I'd wager AI tooling skills are less relevant than knowing your way around a shell. It doesn't help that all of the AI tooling has event more churn than js libraries and package management systems!
I think there's probably some value in understanding how LLMs work, but beyond that it's the same boring skills that matter the most... Critical thinking, design chops, attention to detail, perseverance, etc.
lukan 3 days ago [-]
If you don't get reliable results, but others do, then maybe you should indeed. But your choice.
alienbaby 3 days ago [-]
You know how many pieces of art Damien Hurst creates himself Vs his studio assistants creating them under his direction?
For example, of his 1500 spot paintings, he only actually made 5 of them.
It's not uncommon at all for artists to work this way.
tinfoilhatter 3 days ago [-]
He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.
williamcotton 3 days ago [-]
There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.
tinfoilhatter 3 days ago [-]
Sure - and a factual statement would be, these artists hired assistant artists to help them create their paintings. Just like it would be factual to say that the person directed a LLM to build an app for them.
williamcotton 3 days ago [-]
Sure. So does this diminish the role of Rembrandt? Or is this some semantic black hole you’re drawing us into?
0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory
Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
Aurornis 3 days ago [-]
The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
vel0city 3 days ago [-]
>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
Jcowell 3 days ago [-]
There has to be a better way of sharing config , a profile of sorts you can upload and share
mhitza 3 days ago [-]
Yes, llama.cpp supports presets (an ini format). Though it's not always clear which llama-server flags it does not support.
cjbprime 3 days ago [-]
Does anyone know how to get this working with Claude Code via llama-server? I'm getting a jinja template error about the system prompt not being the first message.
Almondsetat 3 days ago [-]
Have you tried to put the nvidia card into "compute-only mode" since you're not using it for display?
XCSme 3 days ago [-]
With default config via Ollama and 65k context I get 50tps on a 3090.
bilsbie 3 days ago [-]
Any chance I could run it on a GeForce RTX 5060 Ti 16 GB, (64 gb ram)
SwellJoe 3 days ago [-]
You could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.
wgd 3 days ago [-]
Yes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.
metadat 3 days ago [-]
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
qiine 3 days ago [-]
what --no-context-shift is for ?
pulse7 3 days ago [-]
"--context-shift, --no-context-shift ... whether to use context shift on infinite text generation (default: disabled)"
That is obvious. Less obvious is what context shifting does and whether you want it or not.
mrbonner 3 days ago [-]
Jeez, llama.c++ is becoming the ffmpeg cargo cult CLI now
MrDrMcCoy 3 days ago [-]
For the things it does, what tool is better than ffmpeg? Really struggling to see the cargo cult angle. Similar for llama.cpp, as it is literally the only framework I can get to run on my multi-Radeon rig. It is the most portable runtime out there.
hypfer 3 days ago [-]
-funroll-loops
D4Ha 3 days ago [-]
Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop?
If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.
How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
giyanani 3 days ago [-]
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
usagisushi 3 days ago [-]
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
I wonder if turning on GGML_CUDA_NCCL would help reduce delay between cards
evanreichard 3 days ago [-]
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
CamperBob2 3 days ago [-]
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
mips_avatar 3 days ago [-]
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
3 days ago [-]
nullc 3 days ago [-]
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
ericd 3 days ago [-]
Good point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.
bilekas 3 days ago [-]
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
D4Ha 2 days ago [-]
Unfortunately I checked, SLI doesnt work for this situation. Because the program loading the LLM uses CUDA library, which doesnt account for/takes advantage of SLI for this purpose at least.
3 days ago [-]
bmitc 3 days ago [-]
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
hypfer 3 days ago [-]
What makes you say that it would be hard to do that?
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers.
I think at least. I've never tried. My hubris stopped me from trying auto configs.
porphyra 3 days ago [-]
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
solenoid0937 3 days ago [-]
Sure, but front ends like LMStudio exist for that crowd
Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are.
It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
manquer 3 days ago [-]
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
hypfer 3 days ago [-]
> Why would they wish to handcraft this ?
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
manquer 3 days ago [-]
The people OP mentioned about "just want AI" .
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
hypfer 3 days ago [-]
> More and more people are thinking agent native so this is not really a issue.
Why this headache inducing lingo tho? What does that even mean, and why should I sign up for your webinar about that?
manquer 3 days ago [-]
The alternative is to label it AI. That is distasteful for some of us, Intelligence is much stronger than automation in our minds so we avoid the term and try to describe it differently .
bmitc 3 days ago [-]
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
xienze 3 days ago [-]
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
Auracle 3 days ago [-]
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”?
You could override, obviously.
zargon 3 days ago [-]
Yes, llama.cpp does that.
bilekas 3 days ago [-]
I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived.
There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.
Even this command above, it's not clear where op got the model from. So I'm with yah.
For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
MrDrMcCoy 3 days ago [-]
Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.
bilekas 3 days ago [-]
But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page.
If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
MrDrMcCoy 3 days ago [-]
What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?
bilekas 3 days ago [-]
Plain question for you, where can I find the gguf model of this to direct download ?
Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Do you see how steep the barrier for entry is to do anything right ?
unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that, and I want AI to be a better tool.
MrDrMcCoy 3 days ago [-]
> Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
You asked where to find the GGUF files of this model for direct download and I provided it. Almost all useful model files that can be downloaded are hosted on Huggingface.
> They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
I went to Huggingface, went to the Unsloth org, as they tend to be the best, went to the Model page, and went to the "Files and versions" tab.
> Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Unsloth AI is a very popular, highly reputable organization that takes upstream model files, performs some optimization, and provides models in various formats. Apart from speed tweaks, they do not modify the models. They also provide useful benchmarks, copious documentation for local execution, and a Studio application for easy execution and post-training of models.
> Do you see how steep the barrier for entry is to do anything right ?
No. Searching for this information is not difficult. The llama.cpp documentation and guides that Unsloth provide are all you need. Search engines can take you further if you want.
> unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that
Qwen also provides models in GGUF format on Huggingface, but they will not be as performant. Even when first-party GGUFs are available, most people will prefer quants from Unsloth or a few other popular optimizer accounts.
> I want AI to be a better tool.
Best of luck. Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will, and this is as far as I'm willing to carry you.
bilekas 3 days ago [-]
> I went to Huggingface, went to the Unsloth org, as they tend to be the best, went to the Model page, and went to the "Files and versions" tab.
"I went to youtube, gmail, ycombinator, deliveroo, then I went to another site I randomly chose, because they're the best, duh"
Okay.
> Unsloth AI is a very popular, highly reputable organization that takes upstream model files
And How am I supposed to know that arriving to huggingface as a new user? Enlighten me.
> Qwen also provides models in GGUF format on Huggingface
Cool.. Why don't they share em because I genuinely cant find em, I'm dumb.
> Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will
Well, I'm willing, but not from people who I might burn good will. Gracious. You do you.
MrDrMcCoy 3 days ago [-]
> "I went to youtube, gmail, ycombinator, deliveroo, then I went to another site I randomly chose, because they're the best, duh"
The path I described is all within HuggingFace.
> And How am I supposed to know that arriving to huggingface as a new user? Enlighten me.
Because I told you, knowing it was the best starting point for newbies.
> Cool.. Why don't they share em because I genuinely cant find em, I'm dumb.
You could go to Qwen's organization page on HuggingFace, it has a search function at the top, but you would be better served sticking with Unsloth.
> Well, I'm willing, but not from people who I might burn good will. Gracious. You do you.
Expecting others to do everything for you is not the same as trying things and asking questions about what you found.
bilekas 2 days ago [-]
> The path I described is all within HuggingFace.
It's just not. Where do I see the model download ?
The fact we're here is a loss.
It's not intuitive. Deal with it, or fix it.
MrDrMcCoy 2 days ago [-]
I gave you a direct link to where you can get a download, and explained why I chose to link you where I did. I can't post screenshots here drawing you a map of how to use a website. I don't work for Huggingface or any AI company and can affect no changes to how intuitive any of it is, and think is easy enough already.
Huggingface, Unsloth, and llama.cpp all have documentation you can follow that will exceed anything I can tell you here. LMstudio, Lemonade, or Ollama might be even easier for you to use. Take my suggestions or don't.
2 days ago [-]
freehorse 3 days ago [-]
If you do not want to tweak parameters you can just run
llama-server -m model.gguf
That's it
Aurornis 3 days ago [-]
Start by copying the command line from the Unsloth guides.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
bmitc 3 days ago [-]
I had seen this:
> Attention: To be updated for Qwen3
on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not.
I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.
kccqzy 3 days ago [-]
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
CamperBob2 3 days ago [-]
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
acka 3 days ago [-]
While you probably have the audacity to check the commands and code that your favorite LLM generates to complete this task, i'm pretty much certain that at some point in time, due to this pervasive mindset of "just let an AI agent do it," we'll have a front page news story of someone who had their family photos on their NAS uploaded to some public space, their security cameras connected to a "live porn for lolz" webring, and perhaps even their crypto wallets and bank accounts pilfered, all because their favorite LLM got exploited by some prompt injection attack while trying to be helpful doing system administration tasks.
I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.
CamperBob2 2 days ago [-]
It's a risk, all right. I don't even use containers or VMs, I run automated installs on a dedicated server.
Very occasionally I'll have Claude install something on my dev box, but not without close supervision.
bilekas 3 days ago [-]
Promise I'm not being flippant or rude, but why not ask it to write something like a parameterized script to do the same thing for you ?
CamperBob2 3 days ago [-]
Laziness, primarily. Plus, no two models are exactly alike, and some have different instructions in their model card or on the HF page.
bilekas 3 days ago [-]
Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.
2 days ago [-]
2 days ago [-]
3 days ago [-]
parthsareen 3 days ago [-]
Hi! From Ollama here - you can run:
ollama run qwen3.8 (or if on mac qwen3.8:27b-mlx)
naasking 3 days ago [-]
You know free LLMs can help you understand that command line or design your own...
losthubble 3 days ago [-]
just tell claude/codex "set this up on my system $huggingfacelink"
skrebbel 3 days ago [-]
> LM Studio doesn't work behind proxies.
Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet?
I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
bmitc 3 days ago [-]
> I feel like that's a rather niche situation you got there
Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
Aurornis 3 days ago [-]
> Every big company in the world uses a network proxy.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
bilekas 3 days ago [-]
> It's becoming more rare, now.
I would love to see more data on that because I've seen it constantly. There is more isolation maybe where you can do whatever on 'open' network, but always some kind of proxy/vpn connection for hitting anything sensitive.
The operlap is there.. But I would be worried if it was just flat out taken away from secure managed connections just because of AI.. Again, would love to see the numbers of your assumptions.
skrebbel 3 days ago [-]
Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic).
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
vardump 3 days ago [-]
Every big company? YMMV, but I'd say about 20-40% do.
ThreatSystems 3 days ago [-]
If you're on Linux you can probably use proxychains.
spijdar 3 days ago [-]
I wonder how this practically compares with Muse Glimmer, especially quantized.
I've got an RX 7900 XT (20GB of VRAM) and I can run glimmer with a full 128k context window with the draft model at 65-80 tok/s.
This model, on the other hand, I get about 30 tok/s with a 30k context. Raising the context or loading the draft layers for MTP drops performance to 9-15 tok/s.
So I wonder how big the "real world" delta between Glimmer and Qwen is here. I can already run 3-bit DSv4-flash at 9-15 tok/s with 100k~ context, and I suspect it would outperform 4-bit Qwen 3.8 27B here.
I'll have to experiment and see if I just made a stupid mistake somewhere, but it looks like Glimmer might make more sense for the comically specific niche of "20GB VRAM".
jakswa 3 days ago [-]
I'm in the exact same boat with a 7900 XT and a good Glimmer 30B experience. I was really hoping qwen 3.8 would bring some memory/space efficiency savings along the lines of whatever is going on with Glimmer 30B. I have been surprised that a 30 billion model fits and runs better (at higher unsloth quantization! UD-Q4_K_XL fits!) than a 27 billion model.
harhargange 2 days ago [-]
I too purchased the 7900xt as it was cheap with a lot of vram. Qwen 3.6 27b gives me 30 tok/s
marius_ 2 days ago [-]
From my limited tests, Qwen has better reasoning which is a bummer because Muse Glimmer is literally twice as fast.
scirob 2 days ago [-]
We maintain German Langauge index as no one publishes or reruns these sepeartly.
Qwen 3.8 27B is a small improvement with some regressions in our benchmarks not a huge jump like benchmarks listed.
If there are other models (of similar size) out there, that are better at this, please let me know.
_ache_ 2 days ago [-]
From your benchmark, Qwen3.8 is nearer than Opus 4.8 than Qwen3.6.
0.1pp but still.
Also, a lot of people don't really care about german language capacity, maybe people programming in DDP idk.
PS: You benchmark seems saturated. Most values sit @>75% in a benchmark generally indicate that it's no longer as useful as a <70% one. I mean, Qwen3.8 is 77.5% and Fable5 80%, the poll of values is from 65% to 90%.
I tried playing with all the recommended parameters from the unsloth page with no luck...
in one of the high fever ramblings it ended with amen... lol
ssvegeta 2 days ago [-]
I'm seeing this infinite loop behavior during reasoning as well, for the exact same quant from Unsloth. Set presence penalty to 1 and the issue seemly went away, but having a penalty that high worries me for coding tasks. Will try the bartowski one now.
SwellJoe 3 days ago [-]
I'm not seeing that with that exact quantization from Unsloth, so far. I'm seeing a _lot_ of thinking before it starts doing, but it all seems pretty reasonable and not loopy (at least no more loopy than big models, with the expected "But, wait! I need to..." types of back-tracking). So, it's taking a long time, but I don't think it's doing anything pathological.
dofm 3 days ago [-]
I would characterise it as obsessive, not loopy. It's definitely burning through a lot of tokens to ruminate about aspects of tasks that earlier models get done better seemingly on memory.
I still need to understand that.
Setting a reasoning limit does not seem to have good results, because it really seems to go down rabbit holes and that means that cutting reasoning off too early is going to punish the quality on anything it has not got round to pondering yet. But maybe I have to give it a bit more room.
I have not tested in an agentic sense yet, just with my sort of pet queries in LM Studio, but it rather looks like it expects an agentic flow, because telling it that it's a helpful coding agent and changing the order of things in my prompts (telling it up front to ask any clarifying questions before detailing the rest of the prompt) has definitely kept its thinking a bit more on track.
eek2121 3 days ago [-]
Thinking is turned up to max by default. You can turn it down. Unsure why they did this.
nullc 2 days ago [-]
I'm not having any looping.
> --temp 0.2
Looping is a common symptom of changing the sampler settings from what it was RL trained with.
swalsh 3 days ago [-]
WOW, my first try running on my 2 3090's, it was a bit slow... but it FEELS like opus 4.5, i gave it an image and a broad overview of what I wanted it to build, and it built the whole thing from beginning to end.
venusenvy47 3 days ago [-]
For your setup, do you have both 3090's in parallel for the inference of the model?
swalsh 3 days ago [-]
Yes they run in parallel via LMStudio (250k context)
XCSme 3 days ago [-]
Why slow? I see ~50tps on a single 3090
apitman 3 days ago [-]
Yeah make sure you're using MTP and potentially tensor parallelism.
app13 3 days ago [-]
Whats your setup? I have a single 3090 and am struggling to get it purring
XCSme 3 days ago [-]
5900x, 3090 24gb (slightly undervolted), 128gb ddr4, running via Ollama.
I am benchmarking it now locally, will put the results and speed/tps on aibenchy.com
shdh 3 days ago [-]
How did you undervolt the 3090?
XCSme 3 days ago [-]
Msi afterburner, you go around 900mv curve editor, raise it up to normal clock frequency, and it uses like 260w instead 300w for same performance.
There's some YouTube guides for it.
I also undervolted my new 5070ti, same tdp, around 260w instead of 300w and like 8% better performance.
swalsh 3 days ago [-]
yeah, i consider that slow.
XCSme 3 days ago [-]
Oh, ok, that's like the average tps for most AI providers
scrlk 3 days ago [-]
Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch.
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
spmurrayzzz 3 days ago [-]
> They do not beat opus on real-world usage
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
cyanydeez 3 days ago [-]
Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.
spmurrayzzz 3 days ago [-]
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
croemer 3 days ago [-]
Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
spmurrayzzz 3 days ago [-]
Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.
HonshinM 3 days ago [-]
[dead]
enraged_camel 3 days ago [-]
[flagged]
spmurrayzzz 3 days ago [-]
Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
graceful6800 3 days ago [-]
They literally stated in their first sentence that it was coding tasks.
croemer 3 days ago [-]
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
tyre 3 days ago [-]
0% is not 4% less than 4%, that would be 3.84%.
0% is 4 percentage points (pp) less than 4%.
naikrovek 3 days ago [-]
[dead]
KronisLV 3 days ago [-]
> ...but no. They do not beat opus on real-world usage.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
pimeys 3 days ago [-]
If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.
niek_pas 3 days ago [-]
A wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".
valvar 3 days ago [-]
That’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?
xienze 3 days ago [-]
> but then we just need meaningful benchmarks that clearly show that!
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
mlmonkey 3 days ago [-]
In the end, the only benchmark that matters is your own.
bewareofscams 3 days ago [-]
Only useful benchmarks are those you (in particular) don't have access to.
rhdunn 3 days ago [-]
The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.
There are tools like promptfoo designed for this.
metadat 3 days ago [-]
How can you say this when you haven't even tried it yet? Is it just hypothetical vibes?
redox99 3 days ago [-]
Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.
ferrouswheel 3 days ago [-]
But why would you use a model to store factual knowledge, that is stupid. We want intelligence, not a database.
tonis2 3 days ago [-]
Models cant make any decisions if they have 0 idea that the feature exist in this language or in some general fact.
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
ferrouswheel 2 days ago [-]
Implicit knowledge is different from remembering every church that exists in the state of NSW Australia
mirekrusin 3 days ago [-]
That's a good sign.
Ideal local model would not know stuff like who Britney Spears is, best to leave precious weights for something useful.
Of course the line is very blurry but I'd be perfectly happy with local model that doesn't know anything about history, geopolitics, art or even biology etc. just coding, operating systems etc.
Azantys 3 days ago [-]
At one point there were specific -Coder release of Qwen e.g. 2.5 but they dropped that, still wondering how much better 3.6-Coder or 3.8-Coder would be when they ignore everything else
ycui7 3 days ago [-]
that is why we enable web search for the agent. the memory can come from the internet.
deepseek-v4-flash needs web search to return true facts.
In my experience, the facts that are compressed away in small models are ones you don’t need them to memorize. They need familiarity with the essential concepts in a field, so that they will have better comprehension of material put into the context (and make better retrieval decisions). They don’t need to know very particular details which are a hallucination risk and should be derived from the context instead.
willcmcc 3 days ago [-]
There is 0 shot you can make that claim about this model you have not used or downloaded yet
altmanaltman 3 days ago [-]
"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.
kennywinker 3 days ago [-]
If a benchmark is a measure of nothing useful, then model beats model is an objectively useless fact
altmanaltman 3 days ago [-]
But that's the problem with that logic. The author subjectively claims benchmarks are stupid (a debatable opinion) and treats that as inherently true while making the second claim (hence, you shouldn't say model A beat model B).
I can say I find the law stupid, so no one should say person A beat person B in court. But I did not prove the law is stupid; I merely thought it subjectively and demanded others to follow the second part because I believe the first part is true.
Saying that "if the law is useless, court cases are useless" is objectively true and cannot be argued with. But you still need to prove why the law is useless, not why you think it is and even then if people disagree and use the law as a reference, then it's not objectively useless and court cases are not useless as well.
This is the version we'll be testing on our rtx 6000 today! Thank you
eblanshey 3 days ago [-]
Why not just run FP8 on vLLM with that much vRAM? It's plenty fast.
hadlock 3 days ago [-]
For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
eblanshey 3 days ago [-]
I see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?
hadlock 11 hours ago [-]
I didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.
petu 3 days ago [-]
Unsloth one is gguf for llama.cpp (and some other on-device engines).
So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
4chandaily 3 days ago [-]
Run the unsloth if you are using llama.cpp (GGUF)
Run the one you linked if you are running vllm (safetensors)
satvikpendem 3 days ago [-]
Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
danielhanchen 3 days ago [-]
That wasn't our problem right? Gemma officially updated tool calling which we adopted
ycui7 3 days ago [-]
if you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop
Foobar8568 3 days ago [-]
Considering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.
ferrouswheel 3 days ago [-]
Yeah Opus 5 is almost unusable as a daily driver without making me go insane from excessive claude babble.
UncleOxidant 3 days ago [-]
If it's as good as Sonnet 4.6 for most things I'd be happy.
ranguna 3 days ago [-]
Same
WithinReason 3 days ago [-]
I wish each quant was benchmarked on the same tests as the original network so we could compare their performance
It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
cheesecakegood 3 days ago [-]
The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.
zargon 3 days ago [-]
KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)
lostmsu 3 days ago [-]
> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
nullc 2 days ago [-]
It's a cheap to evaluate proxy for totally broken or not, which is a good start.
It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.
lostmsu 2 days ago [-]
> proxy for totally broken or not
You can't really know that either.
xscott 3 days ago [-]
You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
WithinReason 3 days ago [-]
That's not a replacement for benchmarks
cpburns2009 3 days ago [-]
When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.
jrflo 3 days ago [-]
That kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.
CuriouslyC 3 days ago [-]
Qwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.
dannyw 3 days ago [-]
It's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc.
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
jrflo 3 days ago [-]
Interesting! I'll definitely give that a shot then.
alentred 3 days ago [-]
Oh, goody goody goody! Can't wait for a Q4M with *MTP* support. Does anyone know if there are plans for it?
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.
CapsAdmin 3 days ago [-]
It comes with MTP already, they just don't mention it in the filename as it's now the default.
alentred 3 days ago [-]
Oh, thank you for telling me this. Trying it out now.
edg5000 3 days ago [-]
That's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.
Aurornis 3 days ago [-]
In the local LLM communities there is a lot of respect for the Qwen models, but everyone comes to acknowledge that they do a lot of benchmaxxing after using them. Even at full precision they're never as good as models with similar benchmarks.
The $1500 Intel B70 with 32GB of VRAM can run this model at max context with good performance, btw. If you don't want to drop $5-10k for running DeepSeek this is your best budget option for local refactor/small scale dev help
LeBit 3 days ago [-]
I understand the B70 is a bargain vs AMD and especially nVidia offerings, but to me it feels like I would be buying something that would feel too limited in less than a year. 48G would be much more confortable.
And I know the 96G nVidia cards are selling for over 10k$.
The future can’t arrive fast enough!
kamranjon 3 days ago [-]
You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.
Almondsetat 3 days ago [-]
They have terrible cross connectivity, it's not wise to share a model between them
Almondsetat 3 days ago [-]
32GB is perfect for models around 30B parameters. Since qwen has really hit the spot with their 27B dense models, I think it's a good bet. Also, 32GB is enough for other tasks such as image/video generation and loading multiple smaller specialized models
apitman 3 days ago [-]
I wonder how this would stack up against 4x RTX 3060, assuming you have the physical room for them.
Almondsetat 3 days ago [-]
Since the 3060s would have to communicate constantly using pcie, I think I'd go with intel
apitman 2 days ago [-]
Currently software is king when it comes to inference performance though.
Almondsetat 2 days ago [-]
Having the entire model in the vram without having to transfer anything is even more king, though
aappleby 3 days ago [-]
I have a B70, what llama options are you using and what performance are you seeing?
bogzz 3 days ago [-]
Oh, can it work with the /v1/completions/ auto-complete endpoint?
Almondsetat 3 days ago [-]
Sorry, I wrote autocompletion by force of habit. I simply meant it can complete code you have already created a structure for, which personally is very nice
bogzz 3 days ago [-]
I thought so, but thanks for the clarification. I am a little bit disappointed that local autocompletion models have been left by the wayside in favor of models post-trained for agentic coding. Both Codestral and Qwen-2.5-coder are more than a year old at this point, but local auto-complete seems to me to be such a great usecase.
gered 3 days ago [-]
The latest Qwen models (including 3.8 27B) do still support FIM-style in-editor code auto-completion if that's what you're looking for. I wouldn't want to use a large dense model like 27B for such a task (since FIM-style auto-completion really works best with low-latency responses), but it works.
segmondy 3 days ago [-]
You don't need $10k to run DeepSeek, I run it on a $1000 system.
kennywinker 3 days ago [-]
Go on… what is this $1000 system?
jared0x90 3 days ago [-]
4x CMP-170HX he purchased with his time machine last month
758488 3 days ago [-]
Could you elaborate please? Genuinely interested
desterothx 2 days ago [-]
at that price, probably loaded into ddr5 ram, a 3 bit or less quantization, and 3 tps
TomGarden 3 days ago [-]
Any tips on the best approach at running this at an M4 Max 128GB? Token throughput was a bit slow with the last 27B one (MLX), ended up using the A3B variant but if I could get this one to reasonable speed I'd much prefer it.
jwr 3 days ago [-]
I have an M4 Max (unfortunately 64GB). I have been running the Qwen 35B A3B one for a while now, after testing and benchmarking a number of models. That one was consistently the best in class for tasks like despamming, E-mail classification, OCR and dictation post-processing. It was also really fast (90 tokens/s).
I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.
One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.
As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.
jwr 2 days ago [-]
FWIW, Qwen 3.8 27B is actually slower with MTP enabled, unless you generate a lot of code with it (and even then it's not obvious, because you'll likely spend most time thinking).
bredren 3 days ago [-]
How are you using email despamming?
I am looking to automatically recover mail from junk folder due to bad icloud mail filtering.
LoganDark 3 days ago [-]
Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture:
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
minimaltom 3 days ago [-]
What is bpw?
Also whats your cutoff for 'acceptable' speed? I would have said 25tok/s.
LoganDark 2 days ago [-]
Bits per weight.
I consider 'acceptable speed' to be around 150t/s. Why? Well, this is generally what it takes to keep me engaged with the output, rather than immediately switching to other tasks and checking back later. When I check back later, I have more catch-up to do at once, and I haven't been following the process. So I have to recall it, familiarize myself with the new progress, and sort of get back into focus with it, which is a lot of mental work (even if it happens quickly in real-time). I prefer not to have to do this because of how much work it is, so I prefer to watch the agent in real-time and try to follow its reasoning. That also lets me interrupt it quickly when I see it about to make a mistake, or see an important detail I left out.
minimaltom 20 hours ago [-]
Interesting, for anything more than side chats/projects I usually am watching the output generate and thinking about the problem. I have the same issue with switching back, takes a while to recall and page everything back into my context, so I try not to alt-tab away.
rolls-reus 3 days ago [-]
bits per weight
mft_ 3 days ago [-]
Go for a slightly more quantised version, and experiment with different MTP settings. I find that MLX versions are marginally faster on my 64GB M1 Max, but I usually use Unsloth's GGUFs via llama.cpp as there's a much greater range of quants available and I prefer llama.cpp. MTP sometimes also helps a little, but I suspect it's less helpful on my system than others.
This is the way if you need speed. It costs a little bit in smarts, but compare the MTPLX option listed above with the oQ4e-mtp quant using oMLX. The good cacheing layer in oMLX will help things feel faster for some classes of tasks in my experience.
UncleOxidant 3 days ago [-]
Wait for the MTP variants that will likely be out within days. I'm on a 128GB Strix Halo box and for 3.6-27B 8bits I was getting about 9tok/sec (not great). With MTP that gets closer to 18 tok/sec (kind'a usable).
SwellJoe 3 days ago [-]
The MTP is available, but I'm definitely not seeing 18 t/s on the Strix Halo from the 8-bit quantization, even with MTP (more like ~10 with full context on long tasks). This is a slow model (but so was 3.6). What's your exact llama-server command that gets 18 t/s?
anana_ 3 days ago [-]
Seems like MTP is available immediately!
seanmcdirmid 3 days ago [-]
27B is a dense model so it will be slower with an MoE (A3B), but should have better quality? I still haven’t found very good uses cases on my M3 Max for dense models. Even if you can find a MTP version, it doesn’t help much, especially if you compare against an MoE with MTP as well.
brcmthrowaway 3 days ago [-]
Check out MTPLX and limit your context size.
monkmartinez 3 days ago [-]
Qwen3.6-27B has been the main LLM powering my little agentic stack. I have adopted the test and verify approach to any models allowed to run on my machine. When the "heretic" version drops, I will fire up the harness and test. Super excited to see how it stacks up against Qwen3.6!!!
z_rho_one 3 days ago [-]
Beating or comparable to Opus 4.6 in benchmarks. Opus 4.6 was released in February, 2026. So if we still want to talk about a "6 month difference" between Chinese and American AI, the sentence should now be:
Chinese (small model) AI is 6 months behind American (largest model) AI.
jared0x90 3 days ago [-]
Given that glimmer only caught up-ish to 3.6 how far behind is American (largest model) to American (largest model) ?
xlayn 3 days ago [-]
The file "Just loads" on llama.cpp, the Unsloth https://huggingface.co/unsloth/Qwen3.8-27B-GGUF is an MTP file, I see mostly the same speed on pp and generation.
There has to be something wrong with those benchmarks, I find extremely hard to believe a 27B model can work similar or exceed opus 4.6.
cyanydeez 3 days ago [-]
I think you've been drinking the "LLMs only improve by adding parameter counts" that SOTA labs are selling VCs to build data centers so they can keep eating through cash to their own benefits.
To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.
There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.
selectively 3 days ago [-]
[dead]
minimaltom 3 days ago [-]
Worth distinguishing knowledge/task benchmarks from IF / agentic. It doesn't seem out of the question that you can have a small model thats generally good at instruction following and long-horizon agentic, as usually in those cases any requisite knowledge is in the context.
Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.
erdaltoprak 3 days ago [-]
This is one of the most important model releases since most use cases don't need SOTA/Frontier
If the benchmarks are a real indication, we now have a local model that is runnable on a high-end personal PC that trades blows with the leading model Claude Opus 4.6 Max from half a year ago.
Insane if that is the case. Downloading now!
throwaway613746 3 days ago [-]
[dead]
c16 2 days ago [-]
Big thank you to the Qwen team. 3.6 A3B was shocking good, and now I'm hoping they release an 3.8 A3B model too.
Edit: Having used qwen3.8:27b-mlx on MBP M4 64GB, I get around ~45 tok/s. A3B would be great for smaller devices, but it's definitely usable. As I understand it it's a mixture of MLX and MTP.
Thats a huge tok/sec. Prompt prefill is the bottleneck
tosh 3 days ago [-]
27b dense model at Opus 4.6 level
Opus at home
I hope there also will be a new ~10b variant
UncleOxidant 3 days ago [-]
I'm hoping for a 3.8-122B MoE
yassa9 3 days ago [-]
can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example
SwellJoe 3 days ago [-]
I use Gemma 4 12B in the 4-bit quantization for all sorts of vision tasks (image sorting, classification, description). It's also good for the same sorts of things for text (but there are probably better/faster models for text, 12B just happens to excel at vision tasks). The Qwen 9B is also very good for those tasks. If you need to do any kind of "search the web, grab some data, do some kind of action" tasks, these small models are perfect for that. Scraping data in a fuzzy format into a database or report or spreadsheet, producing a dashboard of news, etc.
Small local models can also be used for sub-agent tasks in most agent harnesses. But I'd probably run a larger MoE for that; they're faster and have broader knowledge. The dense models, even very small ones, are not blazing fast.
I don't code with any models small enough to run locally, at least not so far. Qwen 3.8 27B might be the tipping point, though. It's looking really promising, though it's probably slow enough that I won't ever actually use it. I'd rather pay $100/month for a faster model, even if Qwen 3.8 turns out to be smart enough for most of my work. Running it locally with the 8-bit quantization is going at 12-30 t/s, depending on how much context it's chewing on. So, if all you do with AI is coding, then you're better off doing it in the cloud.
But, there's lots of things a small model can do that aren't coding.
tosh 3 days ago [-]
they are all overlapping but:
categorization, information retrieval, semantic search, image description
also with the model as part of an agentic system with tool calling
(edit: it is quite impressive what a small model in a feedback loop can do)
mring33621 3 days ago [-]
9B Qwen models are good and fast for local python coding tasks.
jhonof 3 days ago [-]
Yeah this is the first model I have been able to run locally that actually feels useful, this is unreal I am considering cancelling my claude sub and going to just api (maybe GLM?) for really hard tasks.
apitman 3 days ago [-]
Check out OpenCode Go as well. They give some Kimi K3, Qwen3.8 Max, and GLM5.2 (probably 5.3 soon?) usage which may cover your needs for $10/mo
jhonof 3 days ago [-]
Yeah I was thinking open router but I will look around at options, I genuinely think this model is good enough for like 90+% of my use cases, and the top frontier models are still not that good at architecture so I have to do that myself still so I won't be losing out.
ThouYS 3 days ago [-]
I am so happy right now, qwen3.6-27b was an absolute game changer. To see another one in the same league.. phew
seanmcdirmid 3 days ago [-]
I'm struggling to figure out what to use this for. From the intelligence benchmarks in OMLX. If only they would release another MoE model.
Intelligence Benchmark Comparison
--- Detail ---
Model: scottlowry--Qwen3.8-27B-oQ4e-mtp
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
GSM8K 93.3% 28 30 282 No
MATHQA 46.7% 14 30 26.3 No
HUMANEVAL 96.7% 29 30 156.5 No
MBPP 83.3% 25 30 71.5 No
LIVECODEBENCH 43.3% 13 30 1040.4 No
Model: stamsam--Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-MLX-oQ4-MTP
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
GSM8K 96.7% 29 30 51.9 No
MATHQA 60.0% 18 30 9.1 No
HUMANEVAL 83.3% 25 30 82.9 No
MBPP 80.0% 24 30 29.6 No
LIVECODEBENCH 36.7% 11 30 283.7 No
walrus01 3 days ago [-]
Try manually asking both more discrete esoteric knowledge questions. Or use benchmarks which are less coding focused. The 3.6-35B-A3B with post-training may do well in coding type benchmarks and math but the density of its knowledge falls off in my experience (vs 3.6 27B dense Q8-K-XL unsloth GGUF) when you need to use it for less commonly used domains of knowledge.
seanmcdirmid 3 days ago [-]
My use cases try to avoid accessing world knowledge in the model (I give it access to web search for some adhoc RAG), and ya, I'm just focused on coding so that's the only place I'm looking at right now.
walrus01 3 days ago [-]
I think you may find that the dense 27B also does better if challenged with more rare coding tasks, less common or weird languages or things that aren't well represented in the active 3B parameters of the MoE model (eg: NOT css, javascript, python, c++, etc).
seanmcdirmid 3 days ago [-]
True. Benchmarks have led me astray before, I'll have to actually benchmark it directly in Goose with my real use cases right now (which are mostly writing python code, so it probably still doesn't apply?). Still, I've never noticed a benefit to using a dense model rather than an MoE for coding with an agent tasks.
Balinares 3 days ago [-]
I wonder if Anthropic and OpenAI possibly missed the window to go public. A 27B open-weight model trading blows with the SOTA from just half a year ago is not great news for trillion-dollar investments...
dannyw 3 days ago [-]
Trading blows in some benchmarks is a bit exaggerated. If you try the model, `xhigh` is basically feels like the `max` mode (i.e. massive thinker and extremely presistent), and the amount of world knowledge and intent understanding is nowhere close to an Opus class model even from 6mo ago.
It's still very useful, and it'll probably displace a good bit of API spend; but it's not really "trading blows with SOTA from just a half year ago". A bit overblown on the Anthropic/OpenAI has missed the window I reckon.
Also, on the open weight frontier side, Kimi K3 is pretty expensive, and Deepseek V4 Pro/Flash is getting a little less juicy with price increases.
simjnd 3 days ago [-]
Why would anyone rely on the world knowledge built into a model when the harness can just let it search for current information?
Intent understanding is a big point for sure, but world knowledge I'm not sure I see a use case for it.
Balinares 1 days ago [-]
That's a reasonable point, so let me qualify: at a glance, it seems Qwen 3.8 27B can trade blows with Opus 4.6 on coding tasks, where trading blows doesn't necessarily mean it's a clear winner or even an equal, but does mean it'll at least hold its own and land a punch or two.
(Which I still think is bonkers, FWIW.)
Opus 4.6 is an especially interesting comparison point, I think, because it was a step change; IMO it's when LLMs became serviceable for coding. Yeah, pre-4.6 models did output code, and that code often superficially worked; and bringing it up to production standards still generally meant rewriting it entirely. Opus 4.6 is when that changed.
From my early tests, it's looking like the public benchmarks are not misleading, and Qwen 3.8 somehow got there too, by and large. I've got a few personal tests. One is a mid-complexity one-shot, purposefully underspecified. Beyond a few minor bugs that it could easily fix once pointed out, Qwen 3.8 largely aced it. There are a bunch of things I'd improve, but that was true of Opus 4.6's output too, and by and large the code is clean and well structured. Also worth noting that I'm running Qwen 3.8 fairly aggressively quantized to fit in VRAM; I'd expect tighter results still from the full weights.
Another test I ran is a variant of a common puzzle with an additional structural constraint that makes the usual solution inapplicable, so the model has to actively turn away from the well-known solution and construct a new one that takes the constraint into account. I've never seen a home model pass that test. Kimi K3 passes it, GLM 5.2 passes it (painstakingly). Qwen 3.8 struggles a lot... but does arrive at a correct solution. First time I see a home model do so.
I haven't yet tested it on long multi-turn scenarios. In my experience, that's where pocket models are weakest against heavyweight ones, especially when quantized.
That aside, it does seem like Qwen 3.8 can, in fact, trade blows with Opus 4.6. I don't know yet if it could replace it, and my money would be on no, but I may well be wrong about that considering how weirdly capable it is.
Interestingly, Qwen 3.8's MTP layer is uncannily accurate too. It still gave me good results up to 6 to 8 predicted tokens, which boosts its speed so much it's competitive with Qwen 3.6 MoE. So that's another bizarrely impressive thing about it.
And given all of the above, I do think that the trillions of dollars invested into OpenAI and Anthropic are becoming harder and harder to justify.
jdgoesmarching 3 days ago [-]
Lacking world knowledge is fine for me, I rarely want to rely on the model’s training anyway when there are plenty of great search options to integrate with.
augusto-moura 3 days ago [-]
IMO AI companies are selling the infrastructure and easier access to the models, beyond competing to get the smarter model. We can run Qwen locally, but at what cost? A good enough GPU for running 27B is more than 4,000 USD where I live (Brazil). Even if I were to spend 100s of hours in credits I would never reach a point where the 4000 USD investment pays off. Unless I'm doing it for the sport, building my own LLM rig is not financially viable.
This is without saying that company clients of Anthropic or OpenAI will not have the upfront money to get a RTX 3090 for each and every employee. Much better to sign a deal with the LLM providers and get cheaper token price with a sales person.
amluto 3 days ago [-]
> RTX 3090 for each and every employee
That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).
I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.
voiceeh 3 days ago [-]
What would the time to first token be on a setup like this?
jacquesm 3 days ago [-]
Depending on your compliance needs this may be the only way.
gip 3 days ago [-]
The real target for Anthropic, OpenAI and tons of others are large enterprises and workflows. I think they will do all right if they execute well.
hannofcart 3 days ago [-]
Another way of rephrasing that though is that the public (including large pension funds, retirees, etc.) just dodged a bullet and aren't left holding the bag.
Nevertheless, I doubt anyone's going to emerge unscathed when the valuations start falling though. Vast sections of the American stock market will be wiped out yet and with it the global economy.
culi 3 days ago [-]
Why do you assume 2-3 American companies failing will have a more negative impact on the economy than positive impact of the widespread availability of self-hostable, open-sourced, and efficient models?
There will surely be tons of new companies that make a business off of hosting these models and even tuning them for specific purposes. Not to mention the massive pricing difference will benefit people who actually utilize them
hannofcart 3 days ago [-]
Firstly, I don't think it will be just 2-3 companies that fail. Due the circular investing between companies in the AI ecosystem, the fallout is going to be much bigger.
Secondly, even with that being true, I think open models are a great net benefit for all mankind in the long run.
I think both of those things are true at the same time.
lenerdenator 3 days ago [-]
I mean, that was obvious from the word "go".
Even Zuckerberg isn't as deluded to think that there's money in constantly releasing closed models and hypserscaling their deployment, which is why we got muse-glimmer 30b last week. I have been playing with it on a 32gb MBP w/ M2 Pro. If prosumer-grade hardware from almost three years ago can now offer a novice a way to host a possibly serviceable software development agent, then there's little incentive to pay out for LLMs like Anthropic's or OpenAI's over the long run.
brcmthrowaway 3 days ago [-]
This is why they've been making bank on the secondary market. They can retire now.
seizethecheese 3 days ago [-]
I don’t think so for two reasons:
1. They still have extreme revenue growth (and despite what HN says, good gross margins supposedly).
2. People will pay a premium for the best model. So far, we see catchup but no leapfrogging.
prabhanjana_c 2 days ago [-]
On my RTX3060 - 12GB VRAM + 24GB RAM , with below command
ollama run qwen3.8:27b --verbose "explain mmap”,
I got 2.41 Tokens/s, Not sure if that can be improved considering VRAM doesn’t fit the entire, model.
Architecture thread! Afaict they continue to use gated attention + delta net, which was also adopted+adapted by K3, but im surprised theres no improvements to the residual stream (deepseek are using manifold hyper-connections, kimi have attention residuals) ?
Perf improvements seem to all come from training?
anana_ 3 days ago [-]
As was the case with GLM 5.3, it seems that there is still much juice to be squeezed from post-training
scgopireddy 3 days ago [-]
On M5 Max, I am receiving only 17 tokens/sec.
How do I maximize tok/sec?
akg_67 2 days ago [-]
Are you using MLX version on M5 Max?
qwen3.8:27b-mlx
c7b 3 days ago [-]
For those commenting on the long reasoning, it may be interesting to know that the reasoning effort is set to xhigh by default [0]. Other possible values are medium, low and none. Flag for changing it in llama.cpp below, but note that the long reasoning seems to contribute a great deal to the quality.
The default reasoning is set to 'xhigh', if you want to compare with the past or reduce the time (if you can take the hit in output quality) then you can pick 'high' or 'medium' as well.
mickeyp 3 days ago [-]
Model benchmarks are useful, to a point, but it is the long tail of things you do with the model that determines if it's good at a wide range of activities. Ant/OAI, to their credit, build their models -- even the small ones -- so they follow instructions and do tool calling well, without the system prompts confusing them. This is especially important for long-horizon tool calling.
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
lossolo 3 days ago [-]
Why weren't the points merged again from the "dupe" thread that had 289 points?
What a weird mechanism. If someone is judging a thread/topic/event impact by the number of points it got, then doing this unfairly degrades that thread.
It should have deduped by user and combined the 168(at the time of writing this comment) + 289 points. Just add the twitter link from the previous thread as an additional link in the description, like you normally do, move all the points over, and remove the old thread.
anotherCodder 2 days ago [-]
vllm on 4x 5090 is getting ~20 tok/s with mtp on (their own thread on the hf card). i had qwen3.8-27b up the day after release, one rtx pro 6000, 140 tok/s spec, 0.156s first token, full 262k. image and video on the same api. numbers: https://github.com/avifenesh/memra
try it: https://inference.tiyuvta.ai/app
$0.38 in / $0.20 cache / $2.60 out. openrouter's only host right now is 23 tok/s at $0.45 / $3.20.
anotherCodder 6 hours ago [-]
price update: $0.40 in / $0.10 cache / $2.90 out
anotherCodder 2 days ago [-]
[dead]
natch 3 days ago [-]
Apart from model performance, what harness are people using to come close to Claude Code or Codex workflow styles with tool use, conversations, loops, remote control, etc.?
yalok 3 days ago [-]
and more specifically - what harness is known to be the best fit for Qwen local models, and are there any evals/benchmarks for harness+model pairs?
cloudengineer94 3 days ago [-]
Been trying out Qwen3.8-27B-Q5_K_S_20GB and it's quite interesting it's behaving very well.
Going to give it some coding tasks and see how it goes.
We been eating good at LocalLlama this week.
theanonymousone 3 days ago [-]
I'm wondering whether any provider can offer this for cheaper $/token than the new DSv4 Flash, which is both cheaper and smarter :/
Completely local use is a different story, of course.
kristopolous 3 days ago [-]
q4km is about 48 tps on a 4090. my llama.cpp params are --flash-attn on --parallel 1 --load-mode mmap
devs about to get handed a two 5090 box each and told to max that out
lta 3 days ago [-]
I've been experimenting with a few settings in my 4090 , and if 3.6 run at 90-110 tps, 3.8 staya below 80 tps and it's most often at 60 tps.
I'm using flash attention, mtp speculative decoding (n=2). I've looked at the club-3090 repo, but haven't found anything meaningful but get back to 3.6 performance
xcf_seetan 3 days ago [-]
Hmmm, Just started using it and it is the first time a LLM tell me this: "You can write this yourself. It is not hard." lol
piyh 3 days ago [-]
Qwen 3.6 is ~$2/m tok, 3.8 should be drop in replacement. Gemma 31B is $0.34/m tok. The price differential on these models is massive on openrouter.
jjice 3 days ago [-]
Where do you see that? From what I can see on Open Router, Qwen 3.6 27B (the closest dense equivalent to Gemma 31) is $0.28/m. Am I missing something?
They're comparing Qwen 3.8 Max to Gemma 31B, fundamental mistake.
18 hours ago [-]
piyh 18 hours ago [-]
Your own link says:
In / Out Price
$0.289 / $2.40per 1M
3 days ago [-]
SparkyMcUnicorn 3 days ago [-]
Yeah, I would appreciate if someone could make sense of the pricing differences between these models. How can a provider run DSv4F at lower cost than a 27B dense or 35B A3B model?
Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)?
DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.
SparkyMcUnicorn 3 days ago [-]
If active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)?
dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
petu 3 days ago [-]
> and more memory/computationally expensive to run any way I look at it.
Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B.
Qwen attention is probably more computationaly expensive as well.
Also I guess not much interest/competition in serving Qwen edge model.
mordae 3 days ago [-]
Yeah, it should be basically free. No idea why it is not. I guess KV cache taking up RAM and possibly bad business sense or amortized engineering costs, I honestly do not know.
satvikpendem 3 days ago [-]
Why are you comparing a 2.4 trillion Max model to a 31 billion model?
kanemcgrath 3 days ago [-]
I think I am going to buy a second rtx 3060, as 27B has been just outside of my range for to long, and this looks like the parameter count tipping point
apitman 3 days ago [-]
Running it on 2x3060 now. Works pretty well but VRAM is tight. 4bit quants. 1x128k context, 8bit KV, MTP on.
kanemcgrath 3 days ago [-]
whats the tok/s you get on that. I have heard a few claims of around 30-50 with mtp, but for how cheap the setup is I am surprised I don't hear more about 3060 stacks so I assume there has to be some catch.
apitman 3 days ago [-]
I used GPT-5.6 Sol high to optimize it, and it claimed it was getting 50. I'm seeing ~40 on my goto smoketest: "Make me a vector add in CUDA".
Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.
jamesblonde 2 days ago [-]
For the full weights, unoptimized on vLLM with 2 Nvidia 6000 RTX 48GBs connected by NVLink, i only get 14 tokens/sec with open-code. For batched operations, it climbs to 55 tokens/sec.
For FP8, on a single Nvidia 6000 RTX 48GB, i get 13 tokens/sec on a single GPU and 46 tokens/sec batched.
nullc 2 days ago [-]
On 2x RTX A6000 non-nvlink connected but communicating across the CPU, with llama cpp I get ~60 tok/s for Qwen3.8-27B-UD-Q8_K_XL without any batching.
mraza007 3 days ago [-]
Man what a week,
We just had GLM 5.3 that came out and then we had smaller local model Qwen3.8-27B from Qwen
Just tried using Pi Agent and looks very promising
TomGarden 3 days ago [-]
Really excited to see what people do with this. 3.7 27B was probably the best compromise between size and intelligence to run on consumer hardware
geek_at 3 days ago [-]
Do you mean 3.6 27b? Because qwen 3.7 didn't have an open weight version
sheepscreek 3 days ago [-]
Better than Opus 4.6 at computer use? Comparable with it for SWE? Am I reading this right?
I’ve heard rumours about AI shops optimizing for benchmarks. I also don’t think Qwen/Alibaba would be crazy enough to claim something unless there is some truth in it. Would love to see a side-by-side with Opus 4.6 on categories where Qwen 3.8 27B aces it.
pdude444 3 days ago [-]
Do we really feel like it’s the governments job to regulate OPEN source AI.
At rely health, we use OSS models in a HIPAA complaint and SOC 2 complaint environment to take advantage of asymptotically $0 intelligence to provide best in class care navigation . This should be industry standard -
3 days ago [-]
altruios 3 days ago [-]
remember to let llama.cpp catch up to anything new in this model. Save your judgment until about 2 weeks of use.
chrismartin 3 days ago [-]
'Good' news, there seems to be nothing new architecture-wise. Same as Qwen 3.5 and 3.6, so llama.cpp doesn't know the difference.
synergy20 3 days ago [-]
I wish this can run directly on my RTX 4090, seems like 30B is the sweet spot for dense model to run locally, sadly RTX 5090 is very expensive and I need a new PC and new power supply(and UPS) to run that, adding a second RTX 4090 is another option, but not sure if my PC can do that yet.
baron3dl 3 days ago [-]
even a 3090 will give you the VRAM headroom. i run Q8 on an 3090/A6500 combo. well, Q8 of 3.6-27B. I'm building the Q8 GGUF for 3.8 now, assuming mine will finish before someone else's.
KyleJune 3 days ago [-]
Others in this thread said it runs on RTX 4090.
tristor 3 days ago [-]
I'm hoping to see folks distill this with current generation Opus / Fable reasoning traces. I have had my best results locally so far from Qwopus (Qwen 3.6-27B w/ Opus 4.6 reasoning distilled). This looks GREAT and I am definitely setting this up later today.
kunver 3 days ago [-]
Looks like a pretty significant improvement on the DeepSWE benchmark compared to the previous 27B model.
Anonyneko 3 days ago [-]
Is there any way to turn off thinking if I'm using Ollama? In my particular case, the Ollama API (the software I want no-think for is tied to Ollama's bespoke API). If not, I'll stick to 3.6 for the time being...
3 days ago [-]
SamInTheShell 2 days ago [-]
It's worth running, even quantized. I liked Meta Muse Glimmer's outputs, but qwen3.8-27b@q4_k_s kinda seems way better. Haiku/Sonnet kinda pairing in workflows?
chillaranand 3 days ago [-]
"Generate an SVG of a pelican riding a bicycle" - generated a promising image at first shot.
Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference.
Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.
zazibar 3 days ago [-]
Good news, MTP support is already included in this release. Not sure why they haven't made this clearer.
gaigalas 3 days ago [-]
I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.
greenicon 2 days ago [-]
It's included in the model gguf itself.
gaigalas 2 days ago [-]
Maybe I'm not seeing the speed improvements because it's a dense model, and I'm comparing it to the 3.6 MoE with MTP. Likely, I'm attributing the speed I see to something else.
Honestly, if MTP is available, I don't know why it's so much slower here.
yassa9 3 days ago [-]
Can anyone who has that specific personal test he tries on different models , and tries this model , to tell us here if possible , how good or bad is this new model ? compared to others ?
I only trust those users genuine personal tests
alyandon 3 days ago [-]
There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.
So much potential for that channel. He's got a nice range of tests and a no nonsense presentation style.
However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
alyandon 3 days ago [-]
I noticed he does pay attention to feedback on his videos and I think some people have pointed that out.
yassa9 3 days ago [-]
thaaanks man, this channel seems really informative, although < 10K subs only !
alyandon 3 days ago [-]
It's a relatively new channel - but yeah - I feel the guy puts a lot of effort into what he does and deserves more subs.
macwhisperer 2 days ago [-]
yeah this model is chefs kiss..
running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...
this model--
1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)
2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently
3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)
thanks qwen!! hoping to see the full model range get released...
ThouYS 3 days ago [-]
3.6-27B on little-coder was already mind blowing. looking forward to this guy!
arjie 3 days ago [-]
I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about INT4/AWQ quants for the RTX A6000?
ericd 3 days ago [-]
Was recently thinking about doing something similar, do you basically just have the qwens describe what they see for the flashes?
Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right.
If DSv4 Flash was multimodal, I’d probably be done model shopping for a while
arjie 3 days ago [-]
Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.
Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
ericd 3 days ago [-]
Yeah, I might just replicate what you're doing. Main issue right now is just finding spare vram to actually run another model in parallel... And yeah, if I train up a vision adapter somehow, I'll try to put it up/post about it, seems like we're getting the killer apps for local LLMs right now, where it's just feasible enough if you're enthusiastic enough to be a bit economically irrational, and just useful enough to sort of rationalize.
jlkivey 3 days ago [-]
Note: on the model card the comparison to Opus is Opus 4.6 Max, not 4.7
strong visual reasoning apparently, which is nice. still lacking native audio however. hoping for more companies to embrace the spirit of something like `gemma-4-12b-qat` for actual multi-modality (text, image, video, audio).
cloudengineer94 3 days ago [-]
Having some really good fun with it so far with System Design (ERP) mainly SAP.
I did notice if you go beyond Medium he starts overthinking like hell as per usual for a Qwen model.
davidw 3 days ago [-]
I don't know much about the production of these models. How hard would it be to 'fork' something like this and have it not be full of CCP indoctrination?
regularfry 3 days ago [-]
Look for `heretic` fine-tunes in the next couple of days.
irthomasthomas 3 days ago [-]
Why don't qwen/alibaba host the model themselves? I was looking forward to trying it on their coding plan. Google are the same way with their Gemma models.
spwa4 3 days ago [-]
Pretty sure you can use Gemma models on Google's "Vertex AI".
cmrdporcupine 3 days ago [-]
I found this kind of amusing while running it (using Pi as the harness). Don't know if this is evidence of intense fine tuning from Claude but it smells like it...
" The user wants me to explore the repository at XXXX and report back. Let me start by understanding the project structure, reading the
CLAUDE.md file, and getting a general overview of what this repository is.
Let me start by reading the main project documentation and exploring the directory structure.
I'll take a look around this repo. Let me start by getting a lay of the land.
read resource CLAUDE.md (ctrl+o to expand)
ENOENT: no such file or directory, access 'XXXX/CLAUDE.md'"
naasking 3 days ago [-]
Can anyone confirm whether this new Qwen release is any more concise when thinking? Overthinking was the biggest (only?) downside of the Qwen models.
esotericsean 3 days ago [-]
Need to upgrade to a second 3090! Slowly building up my local models with Krea2, MiniMax H3 (and their new Music3), and now Qwen 3.8
anana_ 3 days ago [-]
Monstrous benchmarks! Hoping it is not benchmaxxed.
sheepscreek 3 days ago [-]
I thought the same. But why claim something so shocking when it can easily be discredited and puts your reputation at risk? If they’re claiming Opus 4.6 level, I expect it to at least match Sonnet 4.6.
g023 3 days ago [-]
All this performance at such small model sizes, why are the API fees so high for the AI monopolists on this side of the world?
jtrn 2 days ago [-]
I have been running it on my M5 Mac and was impressed with how well it worked with Pi coder. It can genuinely work as an assistant fully locally. It helps me configure Dockerfiles, fixed a couple of errors in a test Nuxt app, and so forth. Not very fast at 20 tps (8-bit quant for total memory usage around 30 GB), but enough to feel that I have a true local coding buddy.
Then came the cold water shower. The agent kept trying to figure out a Nuxt icon package issue and was working on it. On the positive side, it was making steady and slow progress without getting stuck in doom loops. But after 20 minutes, I decided to test with Luna. So I switched in Pi and asked it to review the problem and fix it. Same session. Thirty seconds later, it was fully fixed. API cost on open router was $0.02, probably most of it due to the inheritance of the previous session.
At that rate, the power consumption for local would be FAR higher than the API cost to solve the task.
I wish it wasn’t so, but the cost per intelligence is just off the charts now with Luna.
Now I am really liking that GLM 5.3 will probably run fine on 4x DGX Spark. If nothing else, the local models are truly usable for basic coding and assistance. I would have been blown away by the support I could have gotten with Qwen 3.8 when I was starting out coding. Hopefully, the local models will catch up AND the hardware becomes affordable in the future. Local models are keeping the largest LLM providers on their toes.
But right now, it does not make economic or capability sense to run locally. It does make privacy, security, and vendor lock prevention sense, though.
zerologadmin 1 days ago [-]
[flagged]
simonw 3 days ago [-]
Anyone seen this show up in any APIs yet? I'd love to try it out faster than my Mac can run it.
abidlabs 3 days ago [-]
The Hugging Face page (https://huggingface.co/Qwen/Qwen3.8-27B) shows that Featherless supports the model (and includes the inference widget to try it out directly on the page)
brcmthrowaway 3 days ago [-]
This with ddg mcp to fill in world knowledge. Are local models the future when computer architectures catch up?
Valdior 3 days ago [-]
I am waiting for Qwen 3.8 MoE - last time 3.6 MoE was better on codding that just dense 3.6.
scgopireddy 3 days ago [-]
On M5 Max I am only receiving 17 tok/sec, how do I maximize
pu_pe 3 days ago [-]
Seems to be SOTA for its size. Hopefully independent benchmarks will come soon.
my prediction was way too far out. 4.6 at home! Woo.
3 days ago [-]
amazingamazing 3 days ago [-]
With my 5080 laptop so close yet so far to using this stuff
bricss 2 days ago [-]
Barely ~3 t/s on a laptop GPU > . <
expedited123 3 days ago [-]
Kinda was expecting to see Gemma 4 26B in benchmark comparisons :(
kamranjon 3 days ago [-]
Since Qwen 3.6 27b outperforms Gemma 4 26b in most benchmarks I'm not sure the value - also Gemma 26b is a MOE model whereas this is a dense model, so not typically direct competitors at their sizes - Gemma 4 31b comparison would be interesting though.
expedited123 3 days ago [-]
I see! Thanks.
potus_kushner 3 days ago [-]
hopefully for us mere mortals without $4k+ hardware a 35B MOE model will be released. or a new prism ternary bonsai model based on this one.
singingtoday 3 days ago [-]
Played with this on my Mac a bit today. Not bad!
brcmthrowaway 3 days ago [-]
My Strix Halo is about to go overdrive!
Mr_Eri_Atlov 3 days ago [-]
This is the homelab model hands down
brcmthrowaway 3 days ago [-]
Is oMLX or MTPLX supported?
jishnuck26 2 days ago [-]
www.asuralist.in
Couldn’t afford claude pro so I built an web based DSA coach that coaches you on DSA and System design in a socratic way. It uses a qwen 1.5B coder model and inferencing is all done on a CPU. ( who needs a GPU anyway )
fr2029 3 days ago [-]
Will there be an A4B MoE?
webbrain 3 days ago [-]
it is amazing folks! it works like charm with 5090
cakbeslik 2 days ago [-]
smaller and smarter is always better
walrus01 3 days ago [-]
Is it just me or is 3.6 27B Q8 K XL (Unsloth) holding up better in sustained token/s rate as the context fill increases over time? The token/s rate seems to be much higher for a time period deeper into context than previously seen.
At least as compared to 3.6 27B in the same quantization.
crazyemeraldcod 3 days ago [-]
Its so smart!
btbuildem 3 days ago [-]
O joyous day!
imagetic 3 days ago [-]
Yes.
webbrain 3 days ago [-]
wow
ramon156 3 days ago [-]
need another fable uncensored merge with 3.8, really curious what it can deliver
2 days ago [-]
RobertasTa 2 days ago [-]
[flagged]
dude3 3 days ago [-]
[flagged]
fintuner 3 days ago [-]
[flagged]
RobertasTa 3 days ago [-]
[flagged]
sams99 2 days ago [-]
[dead]
scripthound841 3 days ago [-]
[dead]
3 days ago [-]
3 days ago [-]
3 days ago [-]
mmeyerlein 2 days ago [-]
[dead]
steffi_oliver 3 days ago [-]
[flagged]
rohan_tech24 2 days ago [-]
[dead]
kofj 2 days ago [-]
[dead]
zyvop1 2 days ago [-]
[dead]
482937632992 3 days ago [-]
[flagged]
alpha_trion 3 days ago [-]
NICE, i've been waiting for this drop, thanks for posting this
3 days ago [-]
literoldolphin 3 days ago [-]
Why is anyone even using video cards these days? You may as well be burning cash.
This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone:
Those are all for MoE models. And I prefer measuring my tokens in t/s instead of s/t
3 days ago [-]
ferrouswheel 3 days ago [-]
Lol, "burn money on apple hardware instead!"
literoldolphin 3 days ago [-]
And yet it's also a laptop you can basically take anywhere unlike a giant video card with 1000 watt power supply requirements.
bigyabai 3 days ago [-]
Nvidia and AMD both have their own unified memory laptop SOCs, now. Apple Silicon's GPU is relatively weak, it's one of the less-efficient ways to use 100w for compute.
Even the fastest Apple Silicon chips like the M5 Max and the M3 Ultra still put up worse GPU compute performance than last-gen laptop RTX 4080 chips. And they don't scale, the largest M3 Ultra cluster you can configure is still ~2,000x smaller than a DGX SuperPOD. There's a reason Apple discontinued their rackmount hardware, there's very little demand for Apple Silicon in the datacenter.
literoldolphin 2 days ago [-]
As I mentioned above you can't take the data center into the cafe somewhere. We're talking about running local models here.
adrian_b 2 days ago [-]
But you may be able to connect with your laptop to your home server, even from some cafe, and run the LLM remotely.
I always connect back home when I am away and I want access to a beefier computer. On the home server, "Wake on LAN" is enabled, so I can power it on and off from my home router.
Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks.
The VRAM usage seems way less efficient than Gemma 4 or Glimmer though, with 32K of context taking 2.5GB of VRAM. With those, even with MTP or a DFlash model loaded, you could still fit 256k-768k of context. With Qwen 3.8 27B I can't even fit 128k if I quantize V to Q4_0. Maybe with some trial and error I can find some settings that perform well enough with a larger context window that it's still useful for longer tasks.
Lots more testing to do, though I was getting some decent results out of Muse Glimmer which was more than twice as fast and supported huge context windows, managing to solve some bugs that Gemma 4 struggled with. I can't even begin to throw that task at Qwen, because just the prompt alone would use the entire context window and then it would reason for probably that same amount.
If you've got a 32GB card, it should be a decent model even if it really is memory hungry.
EDIT: Tried a few kv cache quantization settings, but it failed with those. I designed this benchmark to be pretty brutal in the face of KLD and any reasoning quality loss, so it's not too surprising. Gemma 4's QAT held up pretty well, at least and could consistently complete it.
What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B?
I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...
Gemma 4 26BA4B would get close, but not quite and sometimes even get stuck in loops despite a repeat penalty.
Do not use any newer updated templates or Unsloth fixes. Use older official templates that released with the models on the huggingface repo. The template here worked: https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0...
llama-server --model "model.gguf" -fa on -np 1 --jinja --ctx-size 262144 -b 768 -ub 768 --cache-type-k f16 --cache-type-v q4_0 --repeat-penalty 1.1 --chat-template-file "chat_template.jinja"
If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario.
My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16.
Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all.
For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used: llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16
i would also like to make one myself for my testing. could you give a rough idea or an outline or point in the general direction on what to do?
Actually perf (speed) is going to mostly on token output, and here Qwen 3.x historically tends to lose badly as it tends to overthink a lot. I'll be running evals on 3.8 myself this weekend to see how its reasoning levels perform.
I assume that AA will have 3.8 numbers soon and Intelligence Index vs Output Tokesn per Intelligence Index Task is a decent way to view that: https://artificialanalysis.ai/models/muse-glimmer?intelligen...
Qwen might be useful to bring out for a second opinion on some more focused details that are largely information complete. Like, use Glimmer to bring together all the relevant critical data and evaluate what the actual problems are, then maybe prototype a solution. If it's still acting up, maybe throw the resulting context at Qwen and let it meditate on it.
I think there was some study done where ideally you would want to throw a bunch of different models at a problem since they don't all have the same perspective or diagnosis on what the problems or the solutions are.
From my testing so far, Qwen 3.8 is better at code but it tends to meander and take forever if it has to look in a lot of places. Glimmer will use like ~1k tokens to formulate a plan and Qwen 3.8 will routinely go over 10k
LM Studio isn't exposing a dropdown for this, at least with the unsloth build.
Unsloth Studio / Desktop does.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
For some usages that speedup more than makes up for it being inferior to Qwen intelligence wise.
Maybe it's implicit that you're using llama.cpp (although you don't mention GGUF), but it's hard to reach concrete conclusions about the model architecture based on one implementation in one runtime.
Llama.cpp checkpoints and moves snapshots of the cache to main RAM for faster resumption after truncation.
What exactly are you doing that the prompt is eating an entire 65536 window? Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing. This allows it to keep the initial instructions in the start of the window and toss out the middle as it goes. IME many people who have written off local models entirely are, for lack of a better term "not holding them right" and consider them worthless.
Definitely also try IQ4_NL for K/V if you haven't. Because it's non linear it's far more hit/miss from model to model and especially quant to quant, I've found generally that it shines brightest when you start with a Q6K+ quant that you otherwise wouldn't bother with because of its size, which it then makes up for in both inference speed and often a larger context.
I do agree about Glimmer, though. It is quite good, far better than the benchmarks let on, especially in heavily agentic cases where it needs to rampage around the OS and utilize many different utilities to zero in on things. It is especially good at being told to try something itself, and if/when it fails, try Qwen, and if Qwen can't do it, call out to Deepseek.
If you set manual MTP settings, you'll override dynamic adjustments the inference engine will try to do. Sometimes the dynamic adjustments aren't optimal. With the settings I use, MTP is always a net win.
> What exactly are you doing that the prompt is eating an entire 65536 window?
I'm not using the full context window.
> Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing.
Tools would not help.
Speaking in terms of wall clock, the expensive part of decode is fetching the weights from memory. Predicting and validating a bunch of tokens using the already fetched weights is insignificant in comparison. Even if you have a poor acceptance rate for predictions, you won't really see a slowdown vs not using MTP.
Want to say more about these private benchmarks? :)
Part of my suspicion is drawn from the thinking trace I got when I tested the car wash problem. That really does seem to have been post-trained; it's too good.
e.g. Gemma 4 26B solves this concisely without adding any filler about fuel economy or how long it will take, but it generally gets there by breaking down the problem in the thinking trace the way you'd expect.
Qwen 3.8 27B is just a little too certain right off the bat in low reasoning mode.
I don't expect you to blab publicly about your private benchmark, but what sorts of reasoning does it require?
#1: it does not require deep world knowledge, because that's not what local models are for.
#2: it directly attacks drive-by understanding, overly linear processing training, poor attention mechanisms, poor reasoning patterns or lazy assumptions that ignore very easy low hanging fruit.
#3: it requires solid instruction following in the face of errors. a lot of models will run into errors and then fall back into some kind of error recovery process that bypasses instruction following.
#4: does not require prompt fine tuning to tweak to each individual model. they all seem to understand.
#5: not unfair. almost every model demonstrates in their reasoning that they have the necessary information that if reasoned about appropriately, could arrive at the correct answer.
#6: not designed to add unnecessary complication that it is intended to exhaust reasoning budgets of any sort, so it is not inherently unfair to models that reason a little more or less. for example, it does not require unnecessary reasoning soaks (ie: hiding the prompt inside base-64 encoding)
#7: has real world use and is probably applicable to overall ability to generalize.
#8: can be scaled up as models get better.
#9: is a very good indicator of how bad a model is falling apart under various inference settings.
I'm not asking anyone to take my word, they can believe or not and in practice people should be taking signals from a variety of places and doing their own testing to see how models behave in their own use cases. What I'm measuring and why I'm measuring it may not be the most important metric for your specific use case.
Most other models are simply failing at these tasks. I think the tasks are relevant to overall model capability, but they are not the only metric. You don't give a jellyfish a tool and expect it to produce wonders, so the other capabilities of the model matter.
Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.
(No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a small chainring.")
I ran that on an M5 Max MacBook Pro using LM Studio and their 17GB GGUF: https://lmstudio.ai/models/qwen3.8
It took 21 minutes(!) and used 22,276 reasoning tokens to produce 3,223 tokens of output.
(For the "they're training on your benchmark now" crowd, all of that cheating didn't prevent it from spending 20 minutes thinking about the task first! You can see the reasoning trace in the link I shared.)
For comparison, here's one I got from qwen3.8-2.4t-a95b on OpenRouter, which is pleasingly animated: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#u...
It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
https://foerstel.com/memory-bikes/
Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.
*Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"
I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.
Also:
> Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.
But, not animated like the Qwen 2.4T one you showed, so minus points for that
https://imgur.com/a/IgYKnmk
Took almost 90 minutes on my Threadripper Pro, and feels well worth it.
28k tokens from the Q8 model
This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working.
It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.
https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...
Some will say conflating flying and flying a kite is bad, but I like it.
Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube).
Consider a series of followup benchmarks.
With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion.
Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark.
Last phase is to iterate with the goal to drive the number of findings to zero.
I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted.
My word! It looks rather decent. Much better than previous efforts.
Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though.
Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).
It's up to you if that counts as benchmaxxed.
SWE: model is strong for its size. It one-shotted the Web app, had no bug. The Rust rewrite only had one bug (reordering didn't work immediately - fixed in one prompt). Committed locally then pushed to my GitHub (https://github.com/DexterLagan/RusTODO). Can't complain. If it can do that reliably, I can use it to make whatever I need.
General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.
Performance on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Perfectly usable for local coding through OpenCode.
Verdict: very nice local and free backup to my usual GPT/Claude/DeepSeek for code. Good for Web searches via Brave search tool calls. What more do you want from a small local model?
I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.
And we are waiting for architectures that do - because it's duly.
A good example of this is planning hardware projects - a larger 200b plus model like DeepSeek V4 flash will recommend parts like motors, real time clocks, voltage regulators etc and it will do so providing exact model names and specifications.
I wouldn’t expect a smaller model to encode all of this information, but it is helpful to understand where that cutoff is because it changes what the model might be useful for. It is a very crude way of measuring because it comes down to the balance of training data at sizes this small - but I do think it conveys something that is helpful in real world tasks.
Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!
Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.
Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).
"Need be helpful concise", "Need maybe not overdo", "Need ask!" Almost caveman.
I have an (unsourced, vague) suspicion that this rather unique thinking trace pattern is actually hobbling the MTP predictions, which seem to perform poorly.
Other notes: it uses the trick of repeating the prompt in the thinking trace.
It also worries about hidden chain of thought appearing in the final answer. It talks about "desired oververbosity 9", which is new. A bit GPT-ish.
It is being extraordinarily thorough in thinking through one of my code requests, but I don't know if the net result will be any better than the 35B MoE.
I asked it to ask me clarifying questions — it did, and it offered me a list of defaults I could simply agree to.
I don't think it is necessarily overthinking in the looping sense, but it is in the being exhaustive sense. I need to explore how it does with a tighter reasoning budget.
I am impressed but I am definitely in Camp Please-35B-A3B-When? here, because on an M1 Max this isn't really practical. I hope they do one, though I think they may not.
All your examples align with this too. It just sounds to me like its optimising by removing the superfluous English words and thinking more Chinesely.
No gender, cases, tenses, conjugation, articles, word order inversion, etc. It's remarkable.
Mandarin has it's own complexities but grammar wise, it feels more optimized or simpler.
Frontier labs have already been doing this for a while, verified in smuggled traces from OAT/Ant.
Simply a way to reduce tokens.
Oh dear. I need to try to understand what is going on here.
ETA: a bit of testing before I climb the wooden hill to Bedfordshire.
LM Studio doesn't seem to display the little dropdown to set reasoning effort, so I bodged the chat template on load to get it to choose 'medium'.
As soon as you switch away from xhigh, it goes back to thinking in normal sentences like Qwen 3.6, rather than in sort of quasi caveman.
And you get all the Wait, Actually, No wait… stuff back.
And it is behaving a lot more like it used to. So that is pretty interesting.
[0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.
But they don't.
Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.
I also don't understand how this repeatedly happens.
My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.
I’m sure there is some basic testing but it might be agentic (LLM likes its own output) and maybe just some human smoke tests.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.
That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.
Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.
Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.
The purpose of all of these is to exclude very unlikely next tokens.
Source: One of the min_p authors
Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
In general it's kind of a benchmaxxing/distillation (I don't mean that pejoratively here, read on) artifact where test time compute's purpose is to refine/zero-in on a particular input:output pair. Basically they're trying to "remember"/re-derive the answer rather than arrive at it deductively - it gets comical/absurd when you see it expending a huge amount of tokens trying to figure out the answer to some trivial question or response to some input, as if it were a trick question or the choice of wording was of the utmost importance. But it's not a bad thing when the output matters or the task is hard: basically the model has been trained to respond/act like a much smarter model and it's probably better for it to overgeneralize that behavior.
Also, IME Gemma and most other local models that support thinking tend to have the same problem, and it's partly only a "problem" because they give you access to the thinking tokens themselves (which you don't in many cases see when working with frontier american models) and you can actually see how they're getting spent. For the local or bulk (runnin on owned or rented hardware) use case where you are paying in time/watts rather than purely by the token, IMO it's a good idea to just look at the task success rate / time per task and not worry about the raw thinking token count.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.
I don't usually rely on AI for much (I'm actually kind of anti-AI, although I follow stuff like this enthusiastically because of the rapid advancements, "average joe" access, and openness), however for what I asked? It was spot on.
The subject was a bit personal, so I won't share. I was just curious what AI would say about the situation and it definitely surprised me, especially since Qwen, while usually great on development/coding stuff, has shown weaknesses in other areas.
Definitely a solid release, and this one runs on my 4090 with minimal loss of quality!
Did I miss something, is it possible to have that model be reliable without thinking?
I used the settings recommended on huggingface/unsloth's model page + whatever suggestion from various LLMs - didn't research too much myself which setting did what.
Ignore whatever nonsense qwen says about what settings to use. Turn off top-p and top-k. Turn on top-n-sigma and keep it at its defaults.
See repetitions that you don't like? Ignore the "repetition, frequency, presence" penalties and turn on DRY (don't repeat yourself).
Want more creativity? Temperature is okay but XTC (eXclude Top Choices) is better.
Llama.cpp quietly has had far advanced samplers compared to the shit stuff Qwen et al recommends for years.
I didn't realise there are people out there unironically using crush
Works better than opencode (pi based) or anything else I have tried for my needs, and by far the prettiest and easiest to reason about what is going on.
But I will bite. What does pi do today better than crush for your use cases?
I like to think of it much like (as a common example most people can relate to) the Newton-Rhapson method for finding roots of a (mathematic) function. Your initial prompt runs, then the ‘harness’ kicks in using whatever methodologies are behind them to iterate on that prompt (back and forth with the model, occasionally with the user to get better guidance) and refine the outputs to hopefully converge back to some sensible output or actions the user was initially looking for.
So you’re hoping for an LLM that sort of ‘zero shots’ or needs minimal iterations from a prompt to give usable results. I find from my anecdata it varies across models and what I’m trying to get it to converge on. I tend to prefer models to not zero shot attempt because they tend to not do great, I want them to get feedback often to let me push them down the route of convergence in spaces I already understand well, meanwhile I like them to explore and give me new paths in spaces I’m not too familiar with.
That’s really what all that “second guessing” is, it’s making sure you’re following a sane path in a massive parameter space of an ambiguously defined problem. Imagine if in Newton’s method you checked the slope and it didn’t decrease from the last iteration and you just say “screw it let’s keep trying that direction.” LLMs and their harnesses tend not to have that base assumption like iteration on decreasing slopes to guide them closer to convergence, it’s a lot messier.
https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B
Not my experience at all. On my M1 Max MacBook Pro, Qwen 3.6 35B-A3B provides better responses in similar time (~60 tokens/second, vs. 50 for Gemma 4 26B-A3B). 3.8 27B is slower, being a dense model, but so are dense Gemmas.
What you describe is a engineering harness problem.
If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck
This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.
Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.
Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.
Edit: Absolutely blazing fast! Getting 163 tokens/sec on WSL and it generated a pretty sweet Pelican.
https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...
I'm sure there's some benchmaxxing going on, and some things you get only with a a larger model.
But I'm feeling pretty confident if not by Gemma 5 than by mid 2028 we'll have local models that are almost always as good as Opus 4.6 was and in many cases far better.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
Fortunately, that isn’t necessary! What LLMs need is a level of fluency with key concepts so that they can (1) make effective use of retrieval tools and (2) understand the material in the context window. 30GB-sized models can absolutely store enough knowledge to do this.
Here is an example from the field of law. Most lawyers who have litigated contract disputes in California know about Civil Code § 1717, which makes any contract providing for attorney fees to a prevailing party mutual, so that even if the contract was written to be one-sided, it won’t be enforced that way. It’s a simple enough concept, but there are many more particulars to it, such as what happens when the fee provision is only written to apply to part of the contract. (Answer: it depends on other facts.)
When a lawyer recognizes that they’re in a situation where § 1717 is relevant, the first thing they will do is pull the statute and read it, because nobody has it memorized. And they don’t need to.
A downside is that you can't just download a lot of that knowledge, vs with the weights the copyright infringement has been outsourced to the lab. Nor can you just search for the info because the internet as a whole is increasingly aggressive at blocking anything that looks like an AI agent.
I'd love to see more retrieval powered local AI-- I think it's an area that open source development could excel. ... but there are advantages of having the knowledge in the weights!
Perhaps what needs happen is for someone to make an "ultrapedia", an AI restatement of a huge library of reference works-- created expressly for the purpose of being a locally stored corpus for AI agents.
...I'm just asking questions here... how sure are we of this?
If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.
If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.
On a more serious note, it depends on your cutoff for "entire human knowledge". It's easy to prove for a generous interpretations of "entire human knowledge" that it can't be done, but hard for something like "all useful human knowledge".
Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB.
I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class <insert name of preferred professional knowledge worker> in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).
Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.
I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.
The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.
My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.
I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".
Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.
Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.
What makes it hard to search for? If you tell me you want to open a lemonade stand in East Bumbling Falls, I won't know that city’s municipal ordinances. But I will know that cities tend to have ordinances. So I’d search broadly for what codes that city has. Then I’d go through each one looking for anything that might affect a lemonade stand.
A strong llm without that knowledge can probably spend millions of tokens parsing every single ordinance and give you an answer if you directly ask about that.
But both a local human or an "all knowing" llm when you just tell it about the stand will be like "hmm wasn't there a law about low calorie beverages being tax exempt? Let me check" and after a very precise search come back with a tip that if you sell sugar free lemonade that's tax exempt.
I mean would you get an accountant from another country just because he can use search?
I’ve looked here: https://github.com/earendil-works/pi/blob/main/packages/codi...
I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).
What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.
It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.
Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.
You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.
Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes.
Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.
If it’s stuck, usually that’s it.
Bigger models “understand” better, both the prompt and the contents. If you will try to read a paper together with a smaller model, the difference is immediately obvious.
Bigger models will “forget” and drift much less.
Compared to closed weight (especially unreleased and access-limited) and open weight/source large sparse MoE LLMs/VLMs, open weight/source small dense models benefits public the most because they just reaches more people.
Compared to Qwen 3.6, 3.8's thinking style changed drastically. With xhigh budget, it thinks a lot MORE, and longer thinking session directly translates to better performance. This tradeoff between performance and computation, memory, etc. is meaningful to me.
However, because Qwen 3.6 and 3.8 share the same architecture, with 32GB vram, llama.cpp, IQ4_XS model, MTP and FP16 mmproj, I can only get 200k context, which is not good compared to 640k context of muse glimmer. Hopefully this problem will be solved in Qwen 4.0 release.
Original images: https://image.non.io/neonRamenDesigns.webp
Qwen 3.8 build: https://html.non.io/neonRamenQwen3.8-27b
Overall I'm very impressed with how well this did. It's a big improvement over 3.6, and it feels on-par with some much, much larger models. I think this one is on-par with Gemini 3.7 Flash.
One thing to note - the build for this on my RTX 6000 pro blackwell took a long time. Easily one of the longest builds I've done. It took around 2 hours to build the site. Obviously we'll have some quants for this soon that will accelerate things, but I was still surprised with how long it took.
Comparison builds from this week:
https://html.non.io/neonRamenGemini3.7
https://html.non.io/neonRamenGLM5.3 (note: non-multimodal)
I'm using kimi-k3 on a real high level dev & analysis task at the moment (with Gemini flash for fast implementation) and it is fantastic.
I think we have passed the point where frontier intelligence is commoditized.
* https://github.com/HarbourMasters/Shipwright/pull/7047
* https://github.com/HarbourMasters/Shipwright/pull/7070
just need to give it clear direction, guide it about how to properly use hooks / preserve decomp code, rewrite all the comments, go back & redo things when it spits out some ugly hack, ...
but this does save time, especially considering this is working with decomp code where many functions are random hex addresses
Fable-1m-max-thinking output, despite being incredibly slow and expensive, feels like it bucked a trend towards superficial loquaciousness in their models that had been building since 4.6.
I'm a bit of a luddite when it comes to upgrading models, fable was the first one to make me give up Opus 4.6-1M-max.
If you look at the hiring marketplace, being just marginally better than your peers can be very lucrative.
If you’re competing on speed or capability as a company (or as an employee), you’re probably going to be willing to pay for the frontier.
I would say that in software this is completely false.
Someone straight out of college, not very useful, makes 75-100K.
Top level senior outside of FAANG is making twice that at best (and at least 10x more capable).
Good enough with a lower price will win the masses
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
[1] The specific directions are "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer." and "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
The MoE models are stupendously faster.
For comparison's sake, Qwen 3.6 35B A3B MLX, same prompt, same machine, thought for 1.59 seconds, used 2,398 tokens, and hit 80.83 tok/s. It actually output just a straight SVG file of an owl (not a bad owl either!), whereas Qwen 3.8 27B spent a zillion times longer gold-plating everything and output HTML+JS+SVG with interactive animation, sound, etc., etc.
Edit: link - https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reason...
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
Tested:
Qwen3.6-35B-A3B-UD-Q4_K_XL
Qwen3.8-27B-UD-Q4_K_XL
27 dense is far more capable than 35A3.
on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
but it failed my basic prompt to compose a vim regex to match CamelCaseWords
downgraded a bit with Q4_K_M from ollama run qwen3.8:27b
and /set nothink and at least 1 regex matched FooBar
prompt eval at 2.8 t/s eval at 0.94t/s
I particularly enjoyed this usage of the regex: /%\1\%/ ... onward for 700+ characters of \%\/ :-)
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B
KAT Coder is another Qwen 3.6 MoE fine-tune that also improves on 3.6 by a measurable amount.
But, I do hope for a bigger MoE, in the 70B to 120B range, something in the Coder Next lineage. I've got a Strix Halo that isn't getting used to its best ability because the best models all run fine on my faster desktop dual 32GB GPU setup, and they run too slow for comfort on the Strix Halo.
Edit: I guess you're right; apparently it's for simulation. I didn't look into it beyond the benchmarks. But, it does work in an agentic context, regardless. It'll write code, and drive an agent.
Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.
I just wouldn't bias myself that way, most people haven't really used local models. This stuff is pretty much all subjective evaluation, there's plenty of reasons for people to favor certain models or disfavor others.
Dude, GLM-5.3 released _today_.
The phrasing "I've settled on" is incorrect for this context.
I think some of these are excited, eager users always ready to hype up the new thing. The same crowd that previously would constantly push for a rewrite from angular->react->svelt->god knows what. Instead now it is on a 6 week cycle and about models/harnesses.
I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
Ultimately, I think some of it is legitimate shifts in who's in lead and what is the best. You gotta dig through a lot of crap to get to that, and I don't really know how to to.
Ultimately I'm saying is that I always applied a fair amount of skepticism about what I see in comment sections but these day it is extreme amounts.
I've assumed the same as well.
I also assume that many of the companies developing these models engage in benchmaxxing.
At my company we've developed our own internal benchmarks for evaluating LLM models as they become available. The benchmarks are tailored to our particular use cases but the utility and knowledge our benchmarks assess is still fairly universally applicable. I see wide differences between what our internal benchmarks report and what the major benchmarks do.
There was a whole lot of fanfare about how amazing GLM 5.2 was when it was released, but it was pure rubbish on our internal benchmark -- far behind OpenAI, Anthropic, Gemini, DeepSeek, etc. I don't know how to reconcile the fact that GLM 5.2 performed very well on some of the major public benchmarks, but consistently performs so poorly on ours. OpenAI models tend to dominate our internal benchmarks.
Then Terminal-Bench 3.0 came out (where the questions are new), and GPT 5.6 Sol received 34.6%, while GLM 5.2 dropped to a whopping 4.6%.
That's odd, since Fable seems to be the leader for the industry. Not cost-effective, but if Anthropic models get dominated by OpenAI in your internal benchmarks, this calls their validity into question. Separately, see the jagged frontier effect. [1]
I've been using GLM 5.2 at my day job (mostly Rust backend work ATM). Nothing that blows away the models from OpenAI and Anthropic, but solidly good enough to get it done. A lot of people have experienced this and the fact that an open weights model can do so is where most of the excitement comes from. Optimizing for benchmarks can only get you so far, and people are quick to criticize models that fall into it (like DeepSeek Pro V4 recently).
[1] https://mitsloan.mit.edu/ideas-made-to-matter/working-defini...
Really no different than when there was suddenly online personas everywhere hyping up TSLA out of the blue. You can see the same thing going on with the BoringCompany subreddit. Crazy that the botnet master isn't able to convince us that Grok is also the best model. I don't think buying twitter was an accident it was probably just literally covering up the evidence.
https://www.rhsmith.umd.edu/research/twitter-bots-boost-tesl...
One thing I have noted a lot more of is that there is comparative fanboying going on. Like "this model has done badly, my favoured competition has a model out soon that will beat this in every way" — comparing a released product to unverifiable hopey claims about an unreleased product.
It's tempting to assume that is bot stuff, but if you've been around any other "hot" technical hobby online (cameras, phones, 3d printers, whatever) you will know it's not. It's just fans aligning into teams, some of them laconic and amusing, some of them overkeen and toxic.
Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?
What you're currently doing is "testing out"
Likewise I have a tool-use eval set and a browser-use eval set. I use frontier models for most interactive tasks that are not home assistant or background agents.
I can imagine someone could build evals for that but I have never done so.
no they aren't. they discontinued their always-sold-out coding plan and launched QwenCloud (basically a friendly frontend with Alibaba Cloud as the hidden backend) and launched typical subscription plans for Qwen & co alongside it.
All of this DeepSeek price increase stuff is related to their specific API, not the entire market.
They were stealing your code for that extremely cheap cached token price.
That was never the real price. Every Chinese LLM API provider is the same to the best of my knowledge.
"CHEAP API" (but pssst we train on every single input aka your code and ideas are ours forever)
If some models are cheaper it's simply because they cost far less to develop. And/or run.
Most of the time Luna figures it out where Deepseek was failing. I rarely have to go to Sol levels.
And yes, OpenAI and Anthropic are highly questionable, too, and so I've ended my OpenAI subscription and I'm moving from Claude to other providers. But even people who do not do that are absolutely justified in drawing a line at supporting Musk, because there's a difference between mere normal shittiess and Musk.
So you get worse outcomes while also financially supporting one of the most prolific pushers of race wars, the destruction of labor rights, environmental protections, and general misinformation.
Truly a win-win?
Contrary to your framing it seems to only reason to use Grok at this point would be ideological.
No one actually cares about the politics as long as the model codes well.
Edit, quite interesting to see the reception to this comment compared to essentially the same type of comment I made on a Grok 4.6 benchmark HN post: https://news.ycombinator.com/item?id=49275385#49275571
It's true that Cursor gives a lot of usage with Grok, most users of Cursor don't care about Musk.
And yeah, people that don't care DO make the world worse through their apathy.
It is not all just 'politics'. Take a stand on some issues. It doesn't cost much not to use Grok.
Some people might not mind (or even know), but I sleep better at night trying to work as ethically as I can.
I do. There are enough models to choose from that I don't need to use one from a guy who did a nazi salute at the presidential podium, and proceeded to rid our government of brown people he felt didn't deserve their job.
> I totally care about politics, especially when it comes to not giving my money to people like musk.
(Btw, mussolini didn’t make the trains run on time)
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much
In "medium" reasoning mode, you get the classic Qwen wait/actually thinking loops you see in 3.6 that I guess will need to be interrupted in the way others do already with an over-thinking guard proxy.
(In one of my test runs it is now on "OK TRULY FINAL APPROACH" after having got through "FINAL FINAL APPROACH". Can relate)
It gets stuck in a thinking loop regarding the WordPress API and (resolvable) ambiguity in my prompt, that I guess might be resolvable with a custom skill with hints on how to look it up (and maybe with the devdocs MCP).
In Low reasoning effort mode it flies through the task and writes pretty solid code. So maybe it is me overthinking what is needed here...
Muse Glimmer thinks well and codes well in my tests; it does fine at this. I really like it so far, but my tests are fairly shallow.
One thing I have been struck by — my prompt includes this sentence:
"Please read the following and then ask me any further clarifying questions you need before proceeding with code generation."
Almost all models I've tested interpret this as an instruction to ask questions regardless. Qwen 3.8 27B is the only one that either expresses confidence that it doesn't need to ask clarifying questions, or in higher reasoning effort ultimately asks questions, but offers up defaults I can choose with a simple reply.
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
https://arxiv.org/abs/2512.11280
"AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference"
Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
https://arxiv.org/abs/2607.05147
But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.
The unweighted average is 4.1
So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.
Also, cool that it carries over from qwen3.6 like that.
Here is the chat template I used (and renamed to qwen.jinja): https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/r...
For image support, drop to NGL 24 and add:
And drop the --no-mmprojArguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.
Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.
So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.
And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.
The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.
Let me return the favor:
Deepseek V4 Flash 0731 Q8
This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.
GGML_CUDA_P2P=1 build/bin/llama-server -m ../DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --alias DeepSeek-V4-Flash --threads 32 --threads-batch 64 --cache-type-k f16 --cache-type-v f16 --metrics --cache-ram 0 --load-mode none --fit on --fit-ctx 1000000 --host 0.0.0.0 -fa on --top-p 0.95 --temp 0.7 --repeat-penalty 1.2 --jinja --tensor-split 1,.8,.1,.15 --cont-batching -b 4096 -ub 4096 -ncmoe 36 --parallel 1 -ot 'blk\.3[67]\.ffn_(gate|up|down)_exps=CPU'
I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.
Searching through Reddit and forums for best commands is annoying.
version = 1
[*] ; Global defaults shared across all presets
threads = 16
[Qwen3.8-27B]
; Model files
model = ..\Qwen3.8-27B-FP8\Qwen3.8-27B-UD-Q4_K_XL.gguf
mmproj = ..\Qwen3.8-27B-FP8\mmproj-BF16.gguf
; Context & compute
ctx-size = 114688
parallel = 1
n-gpu-layers = -1
batch-size = 1024
ubatch-size = 512
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
; Speculative decoding (MTP draft)
spec-default = true
spec-type = draft-mtp
spec-draft-n-max = 5
cache-type-k-draft = q4_0
cache-type-v-draft = q4_0
; Multimodal
no-mmproj-offload = true
; Context behavior
no-context-shift = true
; Chat template & reasoning
jinja = true
reasoning = on
chat-template-kwargs = {"preserve_thinking":"true","reasoning_effort":"medium"}
; Sampling
temp = 1.0
top-n-sigma = 1
top-k = 0
top-p = 1.0
presence-penalty = 0.0
repeat-penalty = 1.0
fit = off
; Auto-load this model when the server starts
load-on-startup = true
[1] https://github.com/ggml-org/llama.cpp/releases
Instructions if you want to do the same:
1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.
2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.
3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.
You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).
Congratulations, you're now running Qwen 3.8 27B.
Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
But maybe someone else has experience to share there
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
Your boss is a human.
This is a computer program running on your PC.
I hope you can see the difference.
Most managers do though?
But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.
Who created it is still me.
None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
[0] https://en.wikipedia.org/wiki/Structure,_sequence_and_organi...
[1] https://en.wikipedia.org/wiki/Abstraction–filtration–compari...
Absurd logic. The AI built the website.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
How many films do you remember the producer above the director?
If there were multiple people involved in the creation of a film I helped to create, I cannot factually say I created it. Just like if someone builds something using code generated by AI, they can't factually say they created it.
For example, most people would agree with this line you wrote: "If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it." A pedant could say "woah hang on, that's ignoring the fact that actually the iPhone is the one recording the images that make up the film, how can the human get all the credit", yet nobody would actually make that argument when discussing who created the film.
Generative AI is pretty much the first time (maybe there are some niche contradictions to this claim?) we consider a tool to be contributing enough creativity to the process that we don't all agree "only one person was operating this tool, so that person is the sole creator" - but some people DO still hold that line, and do consider the human who wrote the prompt to be the creator.
And I don't think there's any objective technical metric we can use to say who's right, it comes down to our collective judgement deciding where the line is.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
It all feels a little like that, again.
Lol, yeah I'd wager AI tooling skills are less relevant than knowing your way around a shell. It doesn't help that all of the AI tooling has event more churn than js libraries and package management systems!
I think there's probably some value in understanding how LLMs work, but beyond that it's the same boring skills that matter the most... Critical thinking, design chops, attention to detail, perseverance, etc.
For example, of his 1500 spot paintings, he only actually made 5 of them.
It's not uncommon at all for artists to work this way.
llms are not human.
$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory
Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
From: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.
How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
For reference, here are also some numbers from my 4060ti + 3060 (16GB + 12GB) setup. [0][0]: https://news.ycombinator.com/item?id=48700091
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Here's my config: https://gitea.va.reichard.io/evan/nix/src/branch/master/modu...
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are.
It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
Why this headache inducing lingo tho? What does that even mean, and why should I sign up for your webinar about that?
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
You could override, obviously.
There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.
Even this command above, it's not clear where op got the model from. So I'm with yah.
For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Do you see how steep the barrier for entry is to do anything right ?
unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that, and I want AI to be a better tool.
You asked where to find the GGUF files of this model for direct download and I provided it. Almost all useful model files that can be downloaded are hosted on Huggingface.
> They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
I went to Huggingface, went to the Unsloth org, as they tend to be the best, went to the Model page, and went to the "Files and versions" tab.
> Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Unsloth AI is a very popular, highly reputable organization that takes upstream model files, performs some optimization, and provides models in various formats. Apart from speed tweaks, they do not modify the models. They also provide useful benchmarks, copious documentation for local execution, and a Studio application for easy execution and post-training of models.
> Do you see how steep the barrier for entry is to do anything right ?
No. Searching for this information is not difficult. The llama.cpp documentation and guides that Unsloth provide are all you need. Search engines can take you further if you want.
> unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that
Qwen also provides models in GGUF format on Huggingface, but they will not be as performant. Even when first-party GGUFs are available, most people will prefer quants from Unsloth or a few other popular optimizer accounts.
> I want AI to be a better tool.
Best of luck. Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will, and this is as far as I'm willing to carry you.
"I went to youtube, gmail, ycombinator, deliveroo, then I went to another site I randomly chose, because they're the best, duh"
Okay.
> Unsloth AI is a very popular, highly reputable organization that takes upstream model files
And How am I supposed to know that arriving to huggingface as a new user? Enlighten me.
> Qwen also provides models in GGUF format on Huggingface
Cool.. Why don't they share em because I genuinely cant find em, I'm dumb.
> Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will
Well, I'm willing, but not from people who I might burn good will. Gracious. You do you.
The path I described is all within HuggingFace.
> And How am I supposed to know that arriving to huggingface as a new user? Enlighten me.
Because I told you, knowing it was the best starting point for newbies.
> Cool.. Why don't they share em because I genuinely cant find em, I'm dumb.
You could go to Qwen's organization page on HuggingFace, it has a search function at the top, but you would be better served sticking with Unsloth.
> Well, I'm willing, but not from people who I might burn good will. Gracious. You do you.
Expecting others to do everything for you is not the same as trying things and asking questions about what you found.
It's just not. Where do I see the model download ?
The fact we're here is a loss.
It's not intuitive. Deal with it, or fix it.
Huggingface, Unsloth, and llama.cpp all have documentation you can follow that will exceed anything I can tell you here. LMstudio, Lemonade, or Ollama might be even easier for you to use. Take my suggestions or don't.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
https://ollama.com/library/qwen3
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
> Attention: To be updated for Qwen3
on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not.
I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.
I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.
Very occasionally I'll have Claude install something on my dev box, but not without close supervision.
Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet?
I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
I would love to see more data on that because I've seen it constantly. There is more isolation maybe where you can do whatever on 'open' network, but always some kind of proxy/vpn connection for hitting anything sensitive.
The operlap is there.. But I would be worried if it was just flat out taken away from secure managed connections just because of AI.. Again, would love to see the numbers of your assumptions.
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
I've got an RX 7900 XT (20GB of VRAM) and I can run glimmer with a full 128k context window with the draft model at 65-80 tok/s.
This model, on the other hand, I get about 30 tok/s with a 30k context. Raising the context or loading the draft layers for MTP drops performance to 9-15 tok/s.
So I wonder how big the "real world" delta between Glimmer and Qwen is here. I can already run 3-bit DSv4-flash at 9-15 tok/s with 100k~ context, and I suspect it would outperform 4-bit Qwen 3.8 27B here.
I'll have to experiment and see if I just made a stupid mistake somewhere, but it looks like Glimmer might make more sense for the comically specific niche of "20GB VRAM".
Qwen 3.8 27B is a small improvement with some regressions in our benchmarks not a huge jump like benchmarks listed.
https://dach.peerbench.ai/compare?models=qwen%2Fqwen3.8-27b,...
German language has never been a big focus for asian models but they still outperform Gemma models https://dach.peerbench.ai/compare?models=openai%2FQwen%2FQwe...
So in production we have been using Gemini Flash Lite as primary and fall back to Qwen when gemini servers are overloaded or just giving us 429
If there are other models (of similar size) out there, that are better at this, please let me know.
Also, a lot of people don't really care about german language capacity, maybe people programming in DDP idk.
PS: You benchmark seems saturated. Most values sit @>75% in a benchmark generally indicate that it's no longer as useful as a <70% one. I mean, Qwen3.8 is 77.5% and Fable5 80%, the poll of values is from 65% to 90%.
that's using llama.cpp
llama-server \ -m ~/somePath/Qwen3.8-27B-UD-Q8_K_XL.gguf \ -np 1 --kv-unified \ -fa on --no-cache-idle-slots --reasoning-preserve \ --temp 0.2 \ --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1 \ --cache-type-k f16 --cache-type-v f16 \ --chat-template-kwargs '{"preserve_thinking": true}' \
I tried playing with all the recommended parameters from the unsloth page with no luck...
in one of the high fever ramblings it ended with amen... lol
I still need to understand that.
Setting a reasoning limit does not seem to have good results, because it really seems to go down rabbit holes and that means that cutting reasoning off too early is going to punish the quality on anything it has not got round to pondering yet. But maybe I have to give it a bit more room.
I have not tested in an agentic sense yet, just with my sort of pet queries in LM Studio, but it rather looks like it expects an agentic flow, because telling it that it's a helpful coding agent and changing the order of things in my prompts (telling it up front to ask any clarifying questions before detailing the rest of the prompt) has definitely kept its thinking a bit more on track.
> --temp 0.2
Looping is a common symptom of changing the sampler settings from what it was RL trained with.
I am benchmarking it now locally, will put the results and speed/tps on aibenchy.com
There's some YouTube guides for it.
I also undervolted my new 5070ti, same tdp, around 260w instead of 300w and like 8% better performance.
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
0% is 4 percentage points (pp) less than 4%.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
There are tools like promptfoo designed for this.
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
Ideal local model would not know stuff like who Britney Spears is, best to leave precious weights for something useful.
Of course the line is very blurry but I'd be perfectly happy with local model that doesn't know anything about history, geopolitics, art or even biology etc. just coding, operating systems etc.
deepseek-v4-flash needs web search to return true facts.
I can say I find the law stupid, so no one should say person A beat person B in court. But I did not prove the law is stupid; I merely thought it subjectively and demanded others to follow the second part because I believe the first part is true.
Saying that "if the law is useless, court cases are useless" is objectively true and cannot be argued with. But you still need to prove why the law is useless, not why you think it is and even then if people disagree and use the law as a reference, then it's not objectively useless and court cases are not useless as well.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
Run the one you linked if you are running vllm (safetensors)
It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.
You can't really know that either.
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.
And I know the 96G nVidia cards are selling for over 10k$.
The future can’t arrive fast enough!
I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.
One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.
As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.
I am looking to automatically recover mail from junk folder due to bad icloud mail filtering.
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
Also whats your cutoff for 'acceptable' speed? I would have said 25tok/s.
I consider 'acceptable speed' to be around 150t/s. Why? Well, this is generally what it takes to keep me engaged with the output, rather than immediately switching to other tasks and checking back later. When I check back later, I have more catch-up to do at once, and I haven't been following the process. So I have to recall it, familiarize myself with the new progress, and sort of get back into focus with it, which is a lot of mental work (even if it happens quickly in real-time). I prefer not to have to do this because of how much work it is, so I prefer to watch the agent in real-time and try to follow its reasoning. That also lets me interrupt it quickly when I see it about to make a mistake, or see an important detail I left out.
Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
This might work for you, but I didn't get on very well with MTPLX when I tried it a while back; YMMV: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized...
Chinese (small model) AI is 6 months behind American (largest model) AI.
To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.
There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.
Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.
If you want Qwen3.8-27B Serving Configs for the DGX Spark vLLM NVFP4 and RTX 4090 llama.cpp GGUF I added the setups here https://x.com/ErdalToprak/status/2088299678085308761?s=20
Insane if that is the case. Downloading now!
Edit: Having used qwen3.8:27b-mlx on MBP M4 64GB, I get around ~45 tok/s. A3B would be great for smaller devices, but it's definitely usable. As I understand it it's a mixture of MLX and MTP.
[1] https://www.reddit.com/r/LocalLLaMA/comments/1voxppd/qwen_38...
Opus at home
I hope there also will be a new ~10b variant
Small local models can also be used for sub-agent tasks in most agent harnesses. But I'd probably run a larger MoE for that; they're faster and have broader knowledge. The dense models, even very small ones, are not blazing fast.
I don't code with any models small enough to run locally, at least not so far. Qwen 3.8 27B might be the tipping point, though. It's looking really promising, though it's probably slow enough that I won't ever actually use it. I'd rather pay $100/month for a faster model, even if Qwen 3.8 turns out to be smart enough for most of my work. Running it locally with the 8-bit quantization is going at 12-30 t/s, depending on how much context it's chewing on. So, if all you do with AI is coding, then you're better off doing it in the cloud.
But, there's lots of things a small model can do that aren't coding.
categorization, information retrieval, semantic search, image description
also with the model as part of an agentic system with tool calling
(edit: it is quite impressive what a small model in a feedback loop can do)
It's still very useful, and it'll probably displace a good bit of API spend; but it's not really "trading blows with SOTA from just a half year ago". A bit overblown on the Anthropic/OpenAI has missed the window I reckon.
Also, on the open weight frontier side, Kimi K3 is pretty expensive, and Deepseek V4 Pro/Flash is getting a little less juicy with price increases.
Intent understanding is a big point for sure, but world knowledge I'm not sure I see a use case for it.
(Which I still think is bonkers, FWIW.)
Opus 4.6 is an especially interesting comparison point, I think, because it was a step change; IMO it's when LLMs became serviceable for coding. Yeah, pre-4.6 models did output code, and that code often superficially worked; and bringing it up to production standards still generally meant rewriting it entirely. Opus 4.6 is when that changed.
From my early tests, it's looking like the public benchmarks are not misleading, and Qwen 3.8 somehow got there too, by and large. I've got a few personal tests. One is a mid-complexity one-shot, purposefully underspecified. Beyond a few minor bugs that it could easily fix once pointed out, Qwen 3.8 largely aced it. There are a bunch of things I'd improve, but that was true of Opus 4.6's output too, and by and large the code is clean and well structured. Also worth noting that I'm running Qwen 3.8 fairly aggressively quantized to fit in VRAM; I'd expect tighter results still from the full weights.
Another test I ran is a variant of a common puzzle with an additional structural constraint that makes the usual solution inapplicable, so the model has to actively turn away from the well-known solution and construct a new one that takes the constraint into account. I've never seen a home model pass that test. Kimi K3 passes it, GLM 5.2 passes it (painstakingly). Qwen 3.8 struggles a lot... but does arrive at a correct solution. First time I see a home model do so.
I haven't yet tested it on long multi-turn scenarios. In my experience, that's where pocket models are weakest against heavyweight ones, especially when quantized.
That aside, it does seem like Qwen 3.8 can, in fact, trade blows with Opus 4.6. I don't know yet if it could replace it, and my money would be on no, but I may well be wrong about that considering how weirdly capable it is.
Interestingly, Qwen 3.8's MTP layer is uncannily accurate too. It still gave me good results up to 6 to 8 predicted tokens, which boosts its speed so much it's competitive with Qwen 3.6 MoE. So that's another bizarrely impressive thing about it.
And given all of the above, I do think that the trillions of dollars invested into OpenAI and Anthropic are becoming harder and harder to justify.
This is without saying that company clients of Anthropic or OpenAI will not have the upfront money to get a RTX 3090 for each and every employee. Much better to sign a deal with the LLM providers and get cheaper token price with a sales person.
That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).
I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.
Nevertheless, I doubt anyone's going to emerge unscathed when the valuations start falling though. Vast sections of the American stock market will be wiped out yet and with it the global economy.
There will surely be tons of new companies that make a business off of hosting these models and even tuning them for specific purposes. Not to mention the massive pricing difference will benefit people who actually utilize them
Secondly, even with that being true, I think open models are a great net benefit for all mankind in the long run.
I think both of those things are true at the same time.
Even Zuckerberg isn't as deluded to think that there's money in constantly releasing closed models and hypserscaling their deployment, which is why we got muse-glimmer 30b last week. I have been playing with it on a 32gb MBP w/ M2 Pro. If prosumer-grade hardware from almost three years ago can now offer a novice a way to host a possibly serviceable software development agent, then there's little incentive to pay out for LLMs like Anthropic's or OpenAI's over the long run.
1. They still have extreme revenue growth (and despite what HN says, good gross margins supposedly).
2. People will pay a premium for the best model. So far, we see catchup but no leapfrogging.
total duration: 8m18.2870918s load duration: 612.105ms prompt eval count: 12 token(s) prompt eval duration: 2.900965s prompt eval rate: 4.14 tokens/s eval count: 1193 token(s) eval duration: 8m14.660618s eval rate: 2.41 tokens/s
System spec: NVIDIA GeForce RTX3060 AMD Ryzen 5 1600 Six-Core B450 AORUS M Mother board. NVIDIA-SMI 620.02 Driver:620.02, CUDA Version: 13.2
I tested it on my 3090, took like 8 hours to benchmark it and my room became a furnace (35+ deg outside temp), but it's really good.
Now, in theory, you can talk directly to your computer and tell it what to do, and it does everything locally.
[0]: https://aibenchy.com/compare/qwen-qwen3-8-27b-medium/z-ai-gl...
Perf improvements seem to all come from training?
qwen3.8:27b-mlx
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
https://news.ycombinator.com/item?id=49299684
What a weird mechanism. If someone is judging a thread/topic/event impact by the number of points it got, then doing this unfairly degrades that thread.
It should have deduped by user and combined the 168(at the time of writing this comment) + 289 points. Just add the twitter link from the previous thread as an additional link in the description, like you normally do, move all the points over, and remove the old thread.
Going to give it some coding tasks and see how it goes.
We been eating good at LocalLlama this week.
Completely local use is a different story, of course.
on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090
make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446
devs about to get handed a two 5090 box each and told to max that out
https://openrouter.ai/qwen/qwen3.6-27b
In / Out Price
$0.289 / $2.40per 1M
Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)?
DeepInfra prices:
Qwen 3.6 27B: $0.32 in / $3.20 out
Gemma 3 27B: $0.08 in / $0.16 out
DeepSeek V4 Flash 0731: $0.08 in / $0.18 out
Qwen 3.6 35B A3B: $0.10 in / $0.95 out
https://openrouter.ai/qwen/qwen3.6-27b
https://openrouter.ai/google/gemma-3-27b-it
https://openrouter.ai/qwen/qwen3.6-35b-a3b
https://openrouter.ai/deepseek/deepseek-v4-flash-0731
dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B.
Qwen attention is probably more computationaly expensive as well.
Also I guess not much interest/competition in serving Qwen edge model.
Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.
For FP8, on a single Nvidia 6000 RTX 48GB, i get 13 tokens/sec on a single GPU and 46 tokens/sec batched.
Just tried using Pi Agent and looks very promising
I’ve heard rumours about AI shops optimizing for benchmarks. I also don’t think Qwen/Alibaba would be crazy enough to claim something unless there is some truth in it. Would love to see a side-by-side with Opus 4.6 on categories where Qwen 3.8 27B aces it.
At rely health, we use OSS models in a HIPAA complaint and SOC 2 complaint environment to take advantage of asymptotically $0 intelligence to provide best in class care navigation . This should be industry standard -
https://avilpage.com/qwen-3.8-27b.html
https://x.com/sgl_project/status/2088281320422322413
Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.
Honestly, if MTP is available, I don't know why it's so much slower here.
I only trust those users genuine personal tests
https://www.youtube.com/@lukesdevlab
I don't know if that is what you are looking for or not and as always your experiences may be different.
However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...
this model--
1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)
2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently
3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)
thanks qwen!! hoping to see the full model range get released...
Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right.
If DSv4 Flash was multimodal, I’d probably be done model shopping for a while
Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
I did notice if you go beyond Medium he starts overthinking like hell as per usual for a Qwen model.
" The user wants me to explore the repository at XXXX and report back. Let me start by understanding the project structure, reading the CLAUDE.md file, and getting a general overview of what this repository is.
Let me start by reading the main project documentation and exploring the directory structure.
I'll take a look around this repo. Let me start by getting a lay of the land.
read resource CLAUDE.md (ctrl+o to expand)
ENOENT: no such file or directory, access 'XXXX/CLAUDE.md'"
Then came the cold water shower. The agent kept trying to figure out a Nuxt icon package issue and was working on it. On the positive side, it was making steady and slow progress without getting stuck in doom loops. But after 20 minutes, I decided to test with Luna. So I switched in Pi and asked it to review the problem and fix it. Same session. Thirty seconds later, it was fully fixed. API cost on open router was $0.02, probably most of it due to the inheritance of the previous session.
At that rate, the power consumption for local would be FAR higher than the API cost to solve the task.
I wish it wasn’t so, but the cost per intelligence is just off the charts now with Luna.
Now I am really liking that GLM 5.3 will probably run fine on 4x DGX Spark. If nothing else, the local models are truly usable for basic coding and assistance. I would have been blown away by the support I could have gotten with Qwen 3.8 when I was starting out coding. Hopefully, the local models will catch up AND the hardware becomes affordable in the future. Local models are keeping the largest LLM providers on their toes.
But right now, it does not make economic or capability sense to run locally. It does make privacy, security, and vendor lock prevention sense, though.
my prediction was way too far out. 4.6 at home! Woo.
Couldn’t afford claude pro so I built an web based DSA coach that coaches you on DSA and System design in a socratic way. It uses a qwen 1.5B coder model and inferencing is all done on a CPU. ( who needs a GPU anyway )
At least as compared to 3.6 27B in the same quantization.
This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone:
https://github.com/drumih/turbo-fieldfare
https://github.com/leonickson1/Swiftlet
https://github.com/sqliteai/warp
Even the fastest Apple Silicon chips like the M5 Max and the M3 Ultra still put up worse GPU compute performance than last-gen laptop RTX 4080 chips. And they don't scale, the largest M3 Ultra cluster you can configure is still ~2,000x smaller than a DGX SuperPOD. There's a reason Apple discontinued their rackmount hardware, there's very little demand for Apple Silicon in the datacenter.
I always connect back home when I am away and I want access to a beefier computer. On the home server, "Wake on LAN" is enabled, so I can power it on and off from my home router.