Rendered at 04:37:24 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
beautiful_apple 4 hours ago [-]
A benchmark table comparing to Qwen 3.5 35B-A3B seems strange when Qwen 3.6 35B-A3B has been out for some time and is significantly better.
I didn't notice the version difference when first reading the article! So this is a heads up to people like me.
ricardobeat 2 hours ago [-]
Their main comparison is 1-bit Bonsai 27B (Qwen3.6 27B) which beats A3B anyway.
walrus01 14 minutes ago [-]
Beats how? In my experience 1-bit bonsai 27B is quite "dumb" when asked a question about a lot of things, as a canned repository of static knowledge from its training dataset. I mean, I literally asked it for a 250 word description of Seattle and it hallucinated a tallest building in the city with an observation deck that doesn't exist, and didn't mention the Space Needle.
Really basic stuff. But then again, the entire thing was running in <6GB of RAM.
But before anyone says 1-bit bonsai 27B beats anything, please actually run it and ask it some questions about topics you already know the answer to.
While Qwen 3.6 35B A3B in Q8 with full context capability (llama-server in no-mmap mode with 262k context will eat 47GB, so not comparable in size either) knows a great deal. The 35B-A3B can even translate multiple pages of English into Farsi and its Farsi output is not far off the quality of what Google Translate does.
“Current approaches to low precision primarily focus on converting models trained in full precision to lower bitwidths. We view this as fundamentally the wrong approach…”
Very excited to see how it performs, I’ve been a bit skeptical of the efficacy of converting existing models - really cool to see one trained from scratch in the ternary format.
walrus01 5 hours ago [-]
I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
walrus01 2 hours ago [-]
I don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang.
It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
getpokedagain 2 hours ago [-]
The schlong test is nearly as funny as drawing shit on bicycles test
walrus01 2 hours ago [-]
I've also been asking LLMs to draw SVGs of literal pelican cases and the results can be more amusing than pelicans on bicycles. You can get pelican cases with cameras, firearms, long cases, square cases, cases that look nothing like pelicans (but more like Zero Halliburton aluminum briefcases), etc. You also get cases that are open or closed depending on the whims of the LLM.
api 3 hours ago [-]
Small or overly quantized LLMs are a genre of humor. Same goes for small image generators. Janky generative AI is like the Geocities web pages of today.
johnsmith1840 3 hours ago [-]
lol I like the first one though. Reads like a great sarcasm response.
I wonder if kids will do this to their parents.
momojo 47 minutes ago [-]
At this point I think Apple just needs to simply not do anything stupid and these small model makers are going to hand them models
arjie 4 hours ago [-]
For small models like this, it’s super important that it works well at tool calling etc. imho because it can’t memorize facts and isn’t big enough to tell when it doesn’t know. I could use it for high quality tool routing or a backup fast model for smaller task set. E.g. I use GPT-5.6 for voice channels at home. I’d prefer to be able to have this do basic tool calls and stuff because of the local speed.
Will give it a crack as a quick model in my clawlike.
sdiazthomas 4 hours ago [-]
This matches what I've seen shipping Apple's on-device model in a Mac app.
The model is reliable at the semantic half. Give it the OCR text of a receipt and it correctly identifies the vendor and the date. What it does not do reliably is follow mechanical instructions. A user asked for dates formatted as TT-MM-JJJJ and got files literally named TT-MM-JJJJ, because it reproduced the format string instead of filling it in. Another asked for uppercase, and the model acknowledged the request in its reasoning and returned lowercase.
The failures were not consistent, which is worse than failing every time. You cannot tell users "this doesn't work", only "this works most of the time", and nobody accepts that from something touching their files.
What fixed it was moving the mechanical part out of the model entirely. The model decides what the document is about; ordinary deterministic code decides how the name is written. Every time I moved that line back toward the model, quality dropped.
Which is a version of your point: with a small model the win isn't making it smarter, it's shrinking what you make it responsible for.
hahahaa 1 hours ago [-]
In mice. I mean Mac Mini M4 not an iPhone.
Also I love AI sites. Fancy font, plain serious style, we "introduce" rather than "release". It's an AI not an animal after all.
getcrunk 3 hours ago [-]
Anyone compare this to ternary bonsai vs the 1 bit
jsphweid 5 hours ago [-]
As of now, 3 of the 5 comments on this page are just 0-1 karma accounts high-fiving the article. Suspicious.
netghost 5 hours ago [-]
You can play with it online. I was pretty impressed with the speed/quality given it's size.
It's definitely not going to replace a larger frontier model, but it's worth keeping an eye on.
Wowfunhappy 5 hours ago [-]
[dead]
Havoc 5 hours ago [-]
Looks promising though much like the bonsai tenary one it hallucinates knowledge quite aggressively. The online chat having search tools covers this up somewhat, but it's still there.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
zooloo99 5 hours ago [-]
Makes me wonder how much we can 'get away with' in terms of raw model capability, when you have tool access for anything specialised or specific.
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
zooloo99 5 hours ago [-]
Edge is edging closer!
Super cool and a taste of what's to come with local AI becoming more accessible to low-end hardware.
I didn't notice the version difference when first reading the article! So this is a heads up to people like me.
Really basic stuff. But then again, the entire thing was running in <6GB of RAM.
But before anyone says 1-bit bonsai 27B beats anything, please actually run it and ask it some questions about topics you already know the answer to.
While Qwen 3.6 35B A3B in Q8 with full context capability (llama-server in no-mmap mode with 262k context will eat 47GB, so not comparable in size either) knows a great deal. The 35B-A3B can even translate multiple pages of English into Farsi and its Farsi output is not far off the quality of what Google Translate does.
I haven't tested something as badly quantized as 35B A3B Q2 which is somewhere around 12GB on disk. https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
Very excited to see how it performs, I’ve been a bit skeptical of the efficacy of converting existing models - really cool to see one trained from scratch in the ternary format.
"please write 250 words on the etymology and history of the word schlong"
https://pastes.io/uhshFgn4
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
chat test link: https://chat.deepgrove.ai/
It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
I wonder if kids will do this to their parents.
Will give it a crack as a quick model in my clawlike.
The model is reliable at the semantic half. Give it the OCR text of a receipt and it correctly identifies the vendor and the date. What it does not do reliably is follow mechanical instructions. A user asked for dates formatted as TT-MM-JJJJ and got files literally named TT-MM-JJJJ, because it reproduced the format string instead of filling it in. Another asked for uppercase, and the model acknowledged the request in its reasoning and returned lowercase.
The failures were not consistent, which is worse than failing every time. You cannot tell users "this doesn't work", only "this works most of the time", and nobody accepts that from something touching their files.
What fixed it was moving the mechanical part out of the model entirely. The model decides what the document is about; ordinary deterministic code decides how the name is written. Every time I moved that line back toward the model, quality dropped.
Which is a version of your point: with a small model the win isn't making it smarter, it's shrinking what you make it responsible for.
Also I love AI sites. Fancy font, plain serious style, we "introduce" rather than "release". It's an AI not an animal after all.
It's definitely not going to replace a larger frontier model, but it's worth keeping an eye on.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
Super cool and a taste of what's to come with local AI becoming more accessible to low-end hardware.