TL;DR

I wanted an LLM that runs comfortably for coding-agent workloads, so I built a quantized model. I use this model with Deepseek Harness and it performs reasonably well. It works with llama.cpp, so give it a try if you’re interested.

Sanshoku-Qwen3.8-3.6-27B-blend-GGUF
huggingface.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF
  • Training
  • Performance
    • 56.8 tok/s on an RTX 4000 Ada (20 GB) with MTP enabled
    • 58.0% on the full SWE-bench Verified, comparable to Ternary-Bonsai-2-27B (60.8%)
    • About 40% less time per task (thanks to loop-suppression tuning)
    • Weaker than Ternary-Bonsai-2-27B on single-shot reasoning and knowledge questions

I ran the whole series of experiments with the power of Claude Code and Codex. My impression is that QAT is quite workable even on a single GPU (B300), as long as you optimize for it.

Same SWE-bench score with less time per task: on the 24 tasks both models resolved, turns per task were 50 vs 52.5 at the same decode speed per request, so the time saved is tokens not generated

Merging Qwen3.8-27B and Qwen3.6-27B

JetBrains published this merge as Qwen3.8-3.6-27B-blend, and I used it as a reference. Qwen3.8 has a tendency to produce long Think blocks, and merging it with Qwen3.6 is a known countermeasure. But quantization brought the long thinking back, so I ended up adding a separate fix for it. (More on that later.)

The merge is a linear interpolation of all tensors in FP32: w36 + 0.5 * (w38 - w36). I put some care into initialization, adjusting the 0.5 coefficient so that the ternary quantization error is smaller at that midpoint between the two teachers. Compared with plain rounding, the relative weight error dropped from 0.44 to 0.28–0.40.

Training method

  • Quantization
    • One scale is shared per 128 weights, and the three values and the scale are decided per group (BitNet b1.58)
    • Each ternary weight is initialized by picking the nearest value in the interval formed by the Qwen3.6 and Qwen3.8 values
    • Attention k/v and the linear-attention alpha/beta stay in FP16 (about 0.7% of all parameters)
  • Training
    • Quantization-aware training (QAT) of the ternary weights via a straight-through estimator
    • Top-k = 32 tokens used for distillation
    • MTP trained at the same time
  • Training data
    • About 1B tokens
    • Code, Japanese and English corpora + agent trajectories + math

The training flow is as follows:

Training round structure

Later on I used pi to try it out for real and see how it behaved, adjusting the training method as I went.

I trained with my own code, using PyTorch, Triton, and flash-linear-attention. I get the speed by optimizing specifically for a single B300 machine.

A training step initially took 83 seconds, but FP8 conversion and kernel fusion brought it down to 28.5 seconds. That’s roughly 9,200 tokens per second, so 1B tokens runs in about 31 hours without stopping.

I generate the teacher’s probability distributions ahead of time and cache them. Generation runs at about 25,700 tokens per second.

Loop suppression

The thing that took the most time was dealing with loops.

An agent works by alternating a turn (a response that issues one command) with its result. A loop is when a turn can’t escape from repeating the same command, and copying is when it transcribes the previous turn’s text verbatim. (“Copying” here is repetition of prior text, not a control-flow loop.)

Models distilled to ternary tend to loop more easily than the original teacher model (Pipis et al. 2025). Once training was complete, the model scored 22/50 on a lightweight evaluation set (Mini) of 50 problems drawn from SWE-bench Verified. 14 of those were loops — 16 if you also count the cases where two or more commands cycle periodically.

I built a small test to measure how hard loops are to escape, then gathered data from contexts where repetition actually happened and added a loss that pushes the escape token’s probability above the repeating token’s (FOCUS & RePAIR).

The first attempt eliminated loops entirely, but the suppression was too strong: the logit difference between the escape token and the repeating token (that is, the log-probability ratio) opened up to 11 nats (1 nat is a factor of e). The number of correct answers stayed flat at 23/50, but on that checkpoint the share of correct answers among submitted patches fell to 59%, and the evaluation score got worse too.

So I raised the required margin to 4 nats and changed the loss so gradients stop once it’s met. I also added data for repetitions inside the thinking block, and that got me to 26/50 with zero loops.

The loss uses only the logit difference between the escape token and the repeating token, at the single position where the loop starts. In a looped context the teacher itself keeps repeating (Mismatch Matters), so the teacher’s answer can’t be used as the target. Instead I set the target to “the escape side is higher by the margin (4 nats)” and use the shortfall as the loss.

The computation is just these two lines (in practice I process multiple positions in one batch).

# At the loop start position, take the logit difference between the escape side and the repeat side.
# The softmax denominator cancels out, so this is directly the log probability ratio (nats)
difference = (h * (w_escape - w_repeat)).sum(-1) / temperature

# If the difference hasn't reached the margin, use it as the loss
loss = F.relu(margin - difference).mean()

For the data, I build contexts that repeat a chunk of a few sentences from the thinking block several times. Even when the same sentence repeats, the tokenizer may split it differently depending on the preceding character, so the token sequence isn’t guaranteed to be identical. So I actually push it through the prompt, then check the periodicity at the token level and discard any sample that breaks. In the prototype, about two-thirds of samples were dropped for this reason.

The next problem was copying. The conversation history keeps the model’s own past thinking verbatim, so as rounds accumulate, copying in long contexts increases; at 32–48k tokens, 66% of turns were copies of the previous turn. When I suppressed that, this time a single turn’s thinking stretched to 7 times its length and the score dropped to 18/50.

After much trial and error, the best weights were the ones that scored 29/50 on SWE-bench Verified Mini with zero loops, so that’s what I published. Most of the 21 failures were timeouts rather than incorrect patches; counting only the tasks where the agent actually submitted a patch, 76% of those patches were correct.

Evaluation

I measured three models (this model, Ternary-Bonsai-2-27B, and Qwen3.8-27B BF16) with the same evaluation tool and the same settings (vLLM, one RTX PRO 6000, effort low). The results are collected in the model card.

I was aiming to beat Bonsai, but overall I lost. I ran the full benchmark only at the end; I should have run comprehensive benchmarks along the way for tuning purposes. I caught up to roughly the same level on SWE, but I feel that better data work is needed.

Compute environment (Nebius)

I rented GPU time from Nebius. The B300 for training cost $4.30/h, and the RTX PRO 6000 for evaluation cost $0.95/h as a preemptible instance. Preemptible instances get killed often, so recovery automation is essential.

Having Claude Code watch things via CronJob was handy, as it automatically recovered from unexpected errors.

Closing thoughts

I’m glad I got a reasonable result.

The starting point was wanting a model that runs comfortably on my local GPU (RTX 4000 Ada), but in hindsight the cloud costs were enough to buy one…?

There’s still a lot I want to try, so I’ll work on it when I have the budget.

Reference notes