What is llm-d?

llm-d is a Kubernetes-native distributed inference framework, built on the vLLM engine, that splits large-language-model serving into separately scaled pools for the compute-heavy prompt-processing phase and the memory-bandwidth-bound token-generation phase. It routes each request toward the replica most likely to hold its relevant cache.