What is LMCache?

LMCache is a library that plugs into large-language-model inference engines to turn the key-value cache generated while processing a prompt into a persistent, shareable resource, instead of a per-request, GPU-only structure that gets discarded. It stores cache chunks across a tiered hierarchy of GPU memory, system memory, disk, and remote storage, so repeated or overlapping prompts can reuse previously computed cache rather than recomputing it.