Can I mix Macs and NVIDIA hardware for AI inference?
Yes, and the two are good at different things. NVIDIA GPUs have more compute, which speeds up reading long prompts (prefill). Macs have more memory bandwidth per dollar, which speeds up writing the answer (decode). Exo Labs measured a 2.8× speedup by prefilling on an NVIDIA DGX Spark and decoding on a Mac Studio.
In that test, Llama 3.1 8B with an 8,192-token prompt took 6.42 seconds on the Mac Studio alone, 4.34 seconds on the DGX Spark alone and 2.32 seconds on both together.
The trick is streaming the prompt’s key-value cache from one device to the other layer by layer, while the next layer is still being computed, so the transfer is hidden.
Sources
Updated