Experimenting with Ternary Quantization for LLMs (Qwen3.8-27B)
TL;DR I wanted an LLM that runs comfortably for coding-agent workloads, so I built a quantized model. I use this model with Deepseek Harness and it performs reasonably well. It works with llama.cpp, so give it a try if you’re interested. Sanshoku-Qwen3.8-3.6-27B-blend-GGUFhuggingface.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF Training QAT (ternary), using a merged model as the teacher, following Qwen3.8-3.6-27B-blend Performance 56.8 tok/s on an RTX 4000 Ada (20 GB) with MTP enabled 58.0% on the full SWE-bench Verified, comparable to Ternary-Bonsai-2-27B (60.8%) About 40% less time per task (thanks to loop-suppression tuning) Weaker than Ternary-Bonsai-2-27B on single-shot reasoning and knowledge questions I ran the whole series of experiments with the power of Claude Code and Codex. My impression is that QAT is quite workable even on a single GPU (B300), as long as you optimize for it. ...