Skip to content
AIBRAINWORX / OpenSourceCoder.comLOCAL-FIRST. BUILT IN THE OPEN.
Inference & Quantization / Text & Coding LLMs

Hugging Face Transformers Streams 180B MoE Experts From SSD on a 32 GB Mac

A new Hugging Face Transformers prototype uses slot-pool caching, speculative router prefetching, and 3-bit expert quantization to run the 180B-parameter Qwen3.8-Flash-Next model on a 32 GB MacBook M5 at 10.0 tokens per second.

Listen to this articleBrowser Voice
Ready
Official company logo or product image for huggingface-transformers-ssd-streams-180b-moe-32gb-mac
Official image or organization logo from huggingface.co; identification only, no endorsement.
Event / context: 2026-11-18. Source check: October 28, 2026.

Streaming 180B MoE Models Into 32 GB of Unified Memory

Running a 180-billion-parameter Mixture-of-Experts (MoE) model on a laptop normally forces a choice between heavy OS swapping or remote cloud inference. In a November 18, 2026 Hugging Face blog post, engineer Tarek Ziadé demonstrated an SSD-streaming architecture inside Hugging Face Transformers that executes the 180B-parameter Qwen3.8-Flash-Next model on a 32 GB Apple MacBook M5. By keeping dense weights in unified memory and streaming quantized experts from solid-state storage into a fixed device cache, the prototype reached 10.0 tokens per second on 200-token decode, coming within 6% of the specialized Strata inference engine on token generation.

The implementation centers on pull request huggingface/transformers#50717 alongside a companion Hugging Face Space (tarekziade/moe-expert-streaming) housing model-specific Metal kernels and conversion scripts. Because routed experts comprise more than 90% of an MoE model's parameters, streaming only active experts lets standard open-source Python tooling run models far larger than available RAM.

Why Existing Offload Paths Stall on MoE Layers

Before modifying Transformers, Ziadé evaluated how existing engines handle MoE weights that exceed accelerator memory. In llama.cpp, memory-mapped GGUF experts can be offloaded to the CPU via flags such as --cpu-moe, --n-cpu-moe, or --fit on, executing matmuls on the CPU so only activations cross PCIe. That works well when experts fit in RAM, but thrashes the OS page cache once weights exceed RAM, while its newer --moe-cache-mib GPU cache splits execution graphs from two per token to 50 and costs roughly 10% overhead even when everything fits.

Elsewhere, vLLM stores offloaded weights in pinned host RAM (--cpu-offload-gb) and reads them over PCIe without caching or a disk tier, forcing hot experts across PCIe on every token even when narrowed via --cpu-offload-params experts. Standard Transformers with Accelerate (device_map="auto") fared worst because forward hooks transfer an entire layer's expert bank to use eight or ten active experts—and on Apple Silicon unified memory, CPU offloading frees nothing. On a 24 GB NVIDIA A10G running Qwen3.6-35B-A3B (with vLLM rerun by Harry Mellor), Strata Q8_0 led at 82–88 tok/s, llama.cpp Q8_0 hit 63–64 tok/s (BF16: 35–36 tok/s), vLLM FP8 reached 20.1 tok/s with expert-only offloading (14.7 under --enforce-eager, 9.7 without expert filtering), vLLM BF16 hit 3.3–7.3 tok/s, and Transformers plus Accelerate managed 0.16 tok/s.

Slot-Pool Caching, Prefetching, and PR #50717

Prototyping on OLMoE-1B-7B, Ziadé found that an SSD-backed LRU cache running one expert per Python loop iteration yielded just 10.1 tok/s compared with roughly 32 tok/s in vanilla Transformers, isolating Python dispatch as the bottleneck. The fix was a unified slot pool: a single preallocated device tensor shared across all layers paired with a per-layer table mapping expert IDs to cache slots. Remapping router-selected expert IDs to slots lets Transformers' existing grouped_mm kernel run all active experts together without custom matmul code, raising OLMoE-1B-7B to 34.6 tok/s versus 38.1 tok/s vanilla.

PR #50717 exposes this path via experts_offload in AutoModelForCausalLM.from_pretrained when device_map maps "experts.*" to "disk". Misses trigger eight I/O threads performing one contiguous read per expert, while the next two layers' routers run early on current hidden states to prefetch likely experts. Fitting Qwen3.8-Flash-Next—48 layers of 512 experts (10 active), totaling 121 GB in FP8—also required handling its 51 GB n-gram embedding table. Offline conversion quantized experts to 3 bits (49 GB on disk, 12 GiB cached on device via cache_bytes), quantized dense weights to 8 bits with MetalConfig(bits=8) (~5.4 GB, +0.2% perplexity over BF16), and kept the n-gram table on SSD for on-demand row reads.

Developer Workflow Impact for Local Coding Agents

For local-first developers, the project demonstrates how close standard Python tooling can get to custom C++ or engine-specific MoE runtimes on laptops. Ziadé noted his own setup pairs a dual-RTX 4090 Windows 11 desktop with an M5 MacBook for portable coding sessions, offloading agent orchestration and autocomplete across machines. Running tarekziade/Qwen3.8-Flash-Next-expert-streaming natively on the laptop removes desktop tethering, using ~18 GiB after load and peaking at 25.4 GiB on a 6K prompt. Still, Ziadé noted that 10.0 tok/s on 200-token decode remains too slow for his daily coding-agent workflows, even after matching Strata within 6%.

Experimental Limits, API Flux, and Hardware Caveats

Key constraints limit near-term production use. First, offline conversion takes roughly five hours, and the 25.4 GiB peak leaves almost no margin on a 32 GB Mac, requiring users to close browsers to avoid macOS swapping. Second, huggingface/transformers#50717 remains an unmerged pull request whose API is still evolving with maintainers; offline quantized experts will likely move into the per-layer quantization framework in #50245 rather than remain a separate argument. Finally, Ziadé disclosed that all code across the PR, Metal kernels, and moe-expert-streaming package was written with AI assistance under his profiling and architectural direction, underscoring its status as an experimental developer release.

Automated Vertex AI (gemini-4-hydrogen) primary-source report; reviewed 2026-11-18.

Follow your curiosity.

Pass the signal.