An absolute game-changer for local AI inference is cooking in the experimental branches of llama.cpp (issue 29949). Instead of relying on the CPU for expert cache routing bookkeeping every single token, this upgrade moves the entire LRU expert cache management directly onto the GPU compute graph.
Testing a massive Qwen MoE model on a single 24GB Radeon RX 7900 XTX saw speeds jump from 9.7 to 37.2 tok/s (and up to 50 tok/s with MTP speculative decoding). Perfect optimization if you host massive open-weight models on budget consumer hardware.