Atomic Chat Runs a 177B Qwen Model on a 64 GB Mac

Atomic Chat released GGUF quantizations for Qwen3.8-Flash-Next, a 177B parameter preview of the Qwen4 architecture featuring Gated DeltaNet, Qwen Sparse Attention, 512 experts, and a 262K context window extensible to 1M. Using an M5 Max system with 64 GB of unified memory, the 85 GB file achieves 36 tok/s with vision components enabled. Atomic Chat crafted a custom imatrix recipe for the release, cutting mean KLD by 63% compared to naive quantization at equal file sizes, and recommends the AD-4.27bpw-Q4_K_M-M64 build running on a llama.cpp build with PR #27742.

The performance breakthrough relies on offloading a massive 51B-parameter, 39 GB n-gram embedding table onto NVMe storage rather than occupying unified memory. For each generated token, the model hashes a three-token sequence to fetch 16 rows of 160 values, transferring roughly 2.7 KB of logical data per forward pass. Because storage pages expand this to an estimated 3 MB/s of actual read traffic, the sub-100-microsecond NVMe access easily fits inside the 28 milliseconds per-token generation budget. This works because the table's read ratio is just 1 parameter in 13 million per token, whereas active mixture-of-experts layers touch 6B parameters and must remain in high-bandwidth memory.

To execute this setup without crashing Apple Silicon, Atomic Chat isolated the n-gram lookup table into shard 2. When llama.cpp maps GPU tensors, interleaving this table with GPU weights causes Metal to wire the table into memory, triggering a kIOGPUCommandBufferCallbackErrorOutOfMemory error during decoding. Preserving shard 2 isolates the table on the SSD while compute-heavy tensors remain in memory.

For machine learning practitioners, this deployment strategy demonstrates how targeted offloading can run massive mixture-of-experts models on hardware previously considered far too small. By offloading static or ultra-sparse components like huge lookup tables to modern high-speed SSDs, developers can run frontier-class architectures locally without incurring prohibitive memory costs.