lmcoleman/Qwen3.6-35B-A3B-BUDGET-GGUF · Hugging Face
huggingface.co
We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Derivative of Qwen/Qwen3.6-35B-A3B, compressed to a hard 13.5 GiB byte budget so that the full model plus 64k context fits in 16 GiB of GPU memory (measured on unified memory; see the fit section for what that does and does not prove). If you have more memory, use the standard tiers instead: MagicQuant hybrids (Q4/Q5/Q6) or the ROCmFPX build.

The experiment:

The smallest standard tier of this model is 20.2 GiB (21.7 GB). It cannot fit in 16 GiB at any context length, so this repo asks a different question: what is the best 35B-A3B you can have if the budget is fixed at 16 GiB total, including 64k of context? (Sizes on this card are GiB, 1024-based, throughout.)

The context arithmetic is what makes it plausible at all. Qwen3.6-35B-A3B is a hybrid-attention model: only 10 of its 40 layers are full attention (2 KV heads x 256 head dim); the other 30 are linear-attention layers whose state does not grow with context. KV cache at 64k is therefore about 1.25 GiB at f16, several times smaller than a dense model of this size. 16 GB minus KV minus runtime buffers leaves roughly 13.5 GiB for weights, a 0.20 size ratio versus BF16, well below the Q4 band.

The weights were fitted to that budget with MagicQuant v2’s size-target search: an exact per-tensor knapsack (753 tensor assignments, imatrix-calibrated) under a hard byte ceiling, verified against real perplexity rather than a proxy. The file landed at 13.51 GiB against a 13.5 GiB request.

A second copy of the model then went through quantization-aware training (QAT): frozen-mode LoRA trained against this exact quantization layout, merged, and re-packed at the identical 753-tensor allocation. Both copies are published because they win on different workloads (measurements below).

  • leanleft@lemmy.mlOPEnglish
    4·
    1 month ago

    for one test, i felt a bit disappointed on this prompt: answer(sample):

    Most software engineering tasks involve much more than just relational data or logical inference. Here’s what Prolog struggles with:
                                                               #### 1. **Data Complexity & Type System**
- Real-world data includes strings, numbers, dates, JSON, XML, images, audio, etc.

    in comparison… every 2B-4B distill model, immediately launched into deeper explorations and answers.

    my evaluation:

    • deceptively good. the answer isnt flashy but actually a decent summary with many subtle,but important, points.
    • prompting matters.
    • smaller models are on par for simple prompts like this. but performance and speed is fantastic. so there is no not much cost difference.