What is Mooncake?

Mooncake is a large-language-model serving architecture that treats the key-value cache, rather than the GPU itself, as the primary resource to schedule. It separates prompt processing and token generation onto independent GPU clusters connected over a network, pooling their spare memory and storage capacity into a shared cache store that both clusters read from and write to.