I tested two local LLMs in two different quantization formats on the same laptop, on the same prompt, three trials each. The result is the opposite of what the marketing says: the supposedly-faster new format lost by 1.8x. Q4_K_M (the older integer-based format) hit 4.7 tokens/second and finished a 200-token generation in 44 seconds. MXFP4 (OpenAI's newer microscaling FP format) hit 2.6 tokens/second and took 71 seconds for the same workload. Same hardware, same prompt, same M-series Mac — the format the new spec was designed for is the one that lost. I went in expecting MXFP4 to win. It didn't. Here's the data, the methodology, and my best guess at why. Why I ran this test I originally wanted to test Q4 vs Q5 vs Q8 of the same model — the classic "does quantization matter for speed" question. The download to get the alternate quant GGUFs would have taken 30-60 minutes over my home network, so I pivoted. Instead, I tested the two models I already had cached on the Mac, each at the quant format they ship with: Alibaba's Qwen3-14B at Q4_K_M and OpenAI's gpt-oss-20B at MXFP4. The pivot turned out to be a better story. I expected MXFP4 to be faster. OpenAI markets gpt-oss-20B as a small, fast, local-first model designed for low-latency inference. MXFP4 is the format they chose for that. The actual numbers said otherwise. What I tested Two local models, both run through Ollama on the same Apple M2 with 24GB of unified memory: Model Quant format Size on disk Qwen3-14B (Alibaba) Q4_K_M (4-bit integer) 9.3 GB gpt-oss-20B (OpenAI) MXFP4 (4-bit microscaling FP) 13.8 GB Q4_K_M is the workhorse of the GGUF ecosystem — the format behind ~70% of local LLM model downloads on HuggingFace in 2026, and the default quant Ollama ships for most models. MXFP4 is newer. It's a microscaling floating-point format designed for NVIDIA Blackwell GPUs and Apple's MLX backend, with better numerical range than integer quants at the same bit count. On paper, MXFP4 should be faster on Apple Silicon because Apple has hardware support for microscaling FP in the M-series Neural Engine. In practice on this Mac, it isn't. The test The prompt was a 263-character code-completion task: "Write a Python function merge_dicts(*dicts) that takes any number of dicts and returns a single dict with all keys merged. If the same key appears in multiple dicts, the later value wins. Include a type hint for the return. Output only the code, no explanation." Expected output: 100-150 tokens of Python. Each model ran the prompt 3 times. I report the first trial separately (cold load) and the average of trials 2 and 3 (warm, resident in memory). Hardware: Apple M2, 24 GB unified memory, Ollama 0.12.8, macOS Tahoe 26.0. Results Model Quant Trial Wall (s) Tok/s Tokens out gpt-oss-20B MXFP4 1 (cold) 165.7 1.8 182 gpt-oss-20B MXFP4 2 (warm) 74.6 2.5 182 gpt-oss-20B MXFP4 3 (warm) 67.0 2.8 182 Qwen3-14B Q4_K_M 1 (cold) 39.1 5.9 200 Qwen3-14B Q4_K_M 2 (warm) 41.9 4.8 200 Qwen3-14B Q4_K_M 3 (warm) 46.2 4.5 200 Warm averages (trials 2 and 3): Model Quant Mean wall Mean tok/s gpt-oss-20B MXFP4 70.8 s 2.6 Qwen3-14B Q4_K_M 44.0 s 4.7 For a 200-token generation, that's a 27-second difference. On a 1,000-token completion at these rates, you're looking at 6 minutes on gpt-oss vs 3.5 minutes on Qwen3. What I think is happening I have a theory, not a proof. Three observations: MXFP4 on Apple Silicon runs through the AMX matrix unit, not the Neural Engine. Apple's AMX supports INT8 and FP16/FP32 natively, with FP4 support added in M4. On the M2, MXFP4 has to be dequantized to FP16 before AMX can do the matmul. The dequantization is a real cost on M2 that disappears on M4. Q4_K_M is a llama.cpp-native format that's been optimized for years. The matmul kernels for it have years of micro-optimization. MXFP4 is newer, less optimized, and dependent on hardware support that M2 doesn't have. gpt-oss-20B's thinking tokens are eating throughput. Both models generated internal "thinking" blocks before the visible response. For 200 visible tokens, gpt-oss generated ~400 total; Qwen3 generated ~250. The thinking tax is bigger on gpt-oss. The combination is a worst-case for MXFP4 on M2. The format is designed for newer hardware (M4, Blackwell) where the dequant cost vanishes and the dynamic-range benefits kick in. On M2, it's strictly slower. Things this changes If you have an M2 or M3 Mac: Q4_K_M and Q5_K_M are still the right picks for speed. They win on every older M-series chip. MXFP4 will catch up on M4. When you upgrade to an M4-class Mac, the dequant cost disappears and MXFP4 should be the fastest format for FP4-class models. For a 14B-class model, Q4_K_M is the sweet spot. 9.3 GB on disk, fits comfortably in 24 GB unified memory, 4.7 tok/s on M2. gpt-oss-20B is not "fast" on M2. The model card claims "designed for local inference" and 20 tok/s, but those numbers are for newer hardware. On M2, you get 2.6 tok/s. Use it for quality, not speed. What I didn't test A few things I didn't measure: Q5 and Q8 of the same model. Original plan was Q4/Q5/Q8 on one model. Skipped because the downloads would have taken 30-60 minutes. The pattern (Q5 ~15% slower than Q4, Q8 ~40% slower) is well-documented. MLX format. Apple has its own model format (MLX) tuned for Apple Silicon. Qwen3-14B in MLX might be faster than Q4_K_M. M3 and M4 Macs. MXFP4's hardware support landed in M4. I don't have an M4 to test on, but the likely outcome is MXFP4 wins there. Other prompt sizes. A 263-character prompt is realistic, but very long contexts (10K+ tokens) stress KV cache differently. Quality. This is a speed test. I did not measure whether Q4_K_M or MXFP4 produces better code. Build it yourself The benchmark lives in the public experiments repo: git clone github.com:Pitambarmahato/hardnumbers-experiments cd hardnumbers-experiments/quant-speed-test python -m venv .venv .venv/bin/pip install -r requirements.txt ollama pull qwen3:14b ollama pull gpt-oss:20b .venv/bin/python src/benchmark.py # Results in results/local_llm_speed_<timestamp>.json The full 6-run benchmark takes about 6 minutes on an M2 24GB. If you re-run on different hardware — especially an M4 Mac — please open a PR with the results. The goal is to make every claim falsifiable, and MXFP4-on-M4 is the next claim that needs testing. Which quant should you actually use Your situation Pick Why M2 or M3 Mac, 16-24 GB unified Q4_K_M or Q5_K_M (GGUF) Fastest, best-supported, llama.cpp-optimized M2 or M3 Mac, 32+ GB unified Q6_K or Q8_0 (GGUF) Quality is better, speed is still fine M4 or later Mac MXFP4 or Q4_K_M — retest MXFP4's hardware support lands in M4 Linux + NVIDIA GPU FP16 or BF16 Tensor cores do the math natively Linux + CPU only Q4_K_M Most CPU-friendly The bottom line: Q4_K_M is the safe pick for Apple Silicon in 2026. It's the fastest format on the hardware most people have. MXFP4 will be the pick once M4-class Macs are common, but that day isn't today. The full writeup with the methodology, the data, and the limitations is at hardnumbers.dev/articles/q4-vs-mxfp4-which-quant-is-faster — that's the canonical version. The interesting question this leaves open is: does MXFP4 actually win on M4, or is the format just slower everywhere? Re-run this benchmark on M4 hardware and we'll know.