I find the tuner script is a bit overly ambitious for VRAM usage, so I run it once to get the initial recommendation, and then I run it again with a --max-mib a few hundred MB lower than that.
Has anyone tried testing with non-recommended share values? I suspect it might be overly ambitious there too, and maybe adding +0.02 or even +0.05 might be a little faster
I guess I’ll be keeping an eye on this https://github.com/ggml-org/llama.cpp/commits/master/scripts/moe-cache-tune.py
It seems like it keeps getting better https://github.com/ggml-org/llama.cpp/pull/31888
can’t wait to revisit this in the next stable release, and hopefully with Qwen 4 if they release a small MoE like the 35b
Sure if compaction was free then I’d do it all the time. One advancement I’d like to see is MoE models that use fewer active parameters/experts when compacting. We already have DeepSeek 4.1 that uses a different number of active parameters depending on what its doing (8b for prefill and 16b for decode) so it doesn’t seem unreasonable to use fewer for compaction. It’d be like using a REAP model, but without having to load the model separately. Or even just use the MTP/draft model for compaction.
Qwen3-Coder?? What year is it?!
Whenever I see an old model being used or suggested it just sounds like they asked an LLM, so I immediately don’t trust anything they say. LLMs love suggesting old models.
7.72 TB, I don’t even have enough SSD storage for this in the entire house lol
if this is real then just the fact that it’s 14T is interesting info about the frontier models
makes me feel like China’s models are at least as efficient as the US frontier models, they’re only behind in hardware
EDIT: this seems to be fake https://huggingface.co/burner22/numbat/discussions/13
this is not a regression just a performance inefficiency
Which model GPU and how much VRAM? 16GB RAM limits your options a bit but maybe LFM 2.5 24B-A2B
If not, Gemma 4 e4b or Ling 3.0 Tiny are about as small and fast as you can get while still being decently smart.
I have 8GB VRAM and 32GB RAM, mostly using llama.cpp to run Tiel 35b at Q4, it’s ok at writing code in an agent as long as it can compile and run tests to iteratively fix its mistakes.
Though there’s been new optimized engines, like Strata, that could maybe allow you to run Qwen Flash Next at decent speeds.
There’s also Qwen 4 coming out soon that may give some good new options for us VRAM poors. Just stick to MoE models.
Usually people don’t use it themselves, you just go on huggingface and search for “Qwen 3.6 35b heretic” or something like that for a premade heretic model.
What are the specs on the computer?
This can happen because of tokenization. Tell it to execute code to do the letter counting and both models will succeed every time
Lol yeah someone tell them about local LLMs, open weights, and abliteration
ANTHROPIC IS CRACKING DOWN ON VIDEO GAME REWRITES 🚨
Claude Code picked up a new rule around Oct 27, sent from Anthropic’s servers under a flag called “tengu_heron_brook”. There’s no setting to turn it off.
The full rule: “Do build original games, games in a genre, and games from the user’s own designs and assets. Don’t build a recreation of a specific existing game, even if it is renamed or called original.”
Is this the end of game rewrites, modding and everything in between?
But it isn’t illegal, so why?
so typical of the American models to be the best at maximizing prices lol
I thought Kimi K3 was funny https://foodtruckbench.com/blog/kimi-k3
It almost seems like it was trying to be as helpful as possible by feeding as many people as it could, too nice for its own good
I agree, I was just pointing out that the issue you quoted earlier is easily avoided
without the possibility of veering off into a tangential rant about the nature of free will.
you can do that with a regular LLM by using structured responses
for home users they would be more palatable if they didn’t require any extra VRAM, if it was just a different “mode” for your normal LLM
it’s so close, I’m gonna Qwum!