Your LLM Repeats the Same Sentence Every Day—10K People Decide to Let It Learn Once

Your LLM inference service repeats the same sentence every day.

A 128K token system prompt? 1000 users send requests, and the cluster recomputes the KV cache for those 128K tokens 1000 times. Each prefill takes 11 seconds, making 1000 times 11,000 seconds of GPU time—and 99% of it is repeated work.

This is not an exaggeration. This is the daily reality of Llama 3.1 70B in a production environment.

Then LMCache says: Each piece of text only needs to be read once by the model.

GitHub 10.1k stars, 1.5k forks, Apache 2.0 license. Created by the University of Chicago, used by Google Cloud, Amazon AWS, NVIDIA, IBM, Oracle, AMD. Deep integration with vLLM and SGLang, officially recommended by NVIDIA Dynamo.

But the engineering cost behind the phrase 'learn once' is far heavier than you think.

KV Cache Is Not a Cache—It's a New Data Object

Most people understand KV cache as: intermediate results temporarily stored in GPU memory during Transformer inference, discarded when the request ends.

LMCache's understanding is completely different. They wrote a blog called "Stop Calling It KV Cache: It's Something Much Bigger"—Stop calling it KV cache, it's much larger than a cache.

In LMCache's architecture, KV cache is a first-class data object:

• Persistent: Does not disappear when the request ends, stored to CPU memory, local SSD, Redis, S3
• Cross-engine reusable: KV cache computed by vLLM instance A can be directly used by instance B
• Compressible: Compressed by CacheGen for transmission, reducing bandwidth usage significantly
• Searchable: KV blocks at non-prefix positions can also be hit (CacheBlend)
• Observable: Cache hit rate, lifecycle, and performance metrics at request and token level, with end-to-end tracing

This is not a cache, this is a **Knowledge Delivery Network (KDN)**—just like CDN distributes web content, KDN distributes knowledge the model has already "learned".

Hierarchical Storage: GPU → CPU → SSD → Redis → S3

LMCache's core architecture is a hierarchical storage system:

When a request comes in, first check L0 (GPU memory). If hit, use directly; if not hit, check L1 (CPU memory), then L2 (local disk), then L3 (Redis), and finally L4 (S3). Each layer has an eviction policy: hot data stays in fast layers, cold data sinks to slow layers.

Redis officially did throughput optimization specifically for LMCache, giving a key SLO: L2 backend needs 2 GB/s throughput to "outrun" prefill—that is, loading KV cache from Redis must be faster than recomputing prefill, otherwise caching is meaningless.

CacheBlend: Breaking the Prefix Cache Constraint

Traditional KV cache reuse has a fatal limitation: only prefixes can be reused.

System prompt + conversation history + RAG retrieval results. If the order of RAG retrieval results changes, even if the content is exactly the same, prefix caching all fails—because the prefix must strictly match from the first token.

This problem is even more severe in Agent scenarios. In each round of conversation, the Agent re-assembles the context: system instructions + history + tool outputs + retrieved documents. The order often changes, so prefix cache hit rate may be only 1/3.

CacheBlend solves this problem.

Its principle: no matter where the KV cache block appears in the input sequence, it can be reused. However, directly concatenating non-prefix KV blocks loses cross-attention information between blocks, degrading generation quality. CacheBlend does selective recomputation—recomputes KV values for tokens near the concatenation points to restore cross-attention, while reusing cache for the rest of the tokens.

Benchmark data:

CacheBlend reduces TTFT by approximately 3x on Llama 70B + 2WikiMQA dataset, increases throughput by 2.8-5x, and achieves F1 Score without degradation.

PD Separation: Prefill and Decode Each Do Their Own Thing

LMCache also supports Prefill-Decode (PD) separation architecture.

Traditional inference executes Prefill and Decode serially on the same GPU. PD separation splits them into different workers:

• Prefill Worker: Dedicated to computing KV cache for input text (compute-intensive)
• Decode Worker: Dedicated to token-by-token generation (memory bandwidth intensive)

After the Prefill Worker computes the KV cache, it is transmitted to the Decode Worker via NVLink, RDMA, or TCP. LMCache abstracts the underlying transmission details through NIXL (NVIDIA's heterogeneous data transfer library), supporting multiple network architectures such as NVLink, RDMA, and TCP/IP.

This allows the Prefill Worker to use large memory GPUs specifically for prefill, and the Decode Worker to use high-bandwidth GPUs specifically for decode, each utilizing its strengths and maximizing resource utilization.

Four Major Solutions Cross-Evaluation: Who Solves What Problem

KV cache management is becoming the infrastructure layer for LLM inference, and LMCache is not the only one working on it.

LMCache's core differentiation is vendor-neutral + non-prefix reuse + KV compression. It is not tied to any specific engine or storage; users can freely switch between vLLM and SGLang, and freely choose Redis, S3, or Mooncake as the backend.

NVIDIA Dynamo's advantage is deep binding with NVIDIA hardware ecosystem, with optimal NVLink + RDMA transmission performance. Mooncake is Kimi's self-developed solution, deeply optimized for its own business. Huawei UCM is a component of Huawei AI Factory, not open source.

Clear Costs: Cache is Not a Free Lunch

1. Storage backend selection is the first hurdle.

LMCache supports 8 storage backends, but each has its applicable scenarios:

• CPU DRAM: Lowest latency, but limited capacity, cannot be shared across multiple nodes
• Local SSD: Large capacity, but bound to a single node; if the node fails, the cache is lost
• Redis/Valkey: Can be distributed and shared, but requires 2 GB/s throughput SLO, not all Redis deployments can achieve this
• S3: Unlimited capacity, but latency above 100ms, only suitable for cold data
• NIXL/GDS: Best performance, but requires specific NVIDIA hardware

Choosing the wrong backend makes cache loading slower than recomputing, actually slowing down inference.

2. Cache consistency is a classic problem in distributed systems.

Multiple vLLM instances share the same KV cache pool. When one instance updates the cache, how do other instances know? LMCache currently manages cache metadata via the Controller component, but under high concurrency and large-scale deployments, the complexity of cache invalidation and consistency grows exponentially.

3. Non-prefix reuse has quality risks.

CacheBlend's selective recomputation maintains F1 Score without degradation in most scenarios, but under extreme scenarios (tasks heavily dependent on cross-block attention), there may still be minor quality loss. The evaluation datasets in the paper are limited (2WikiMQA), and production workloads vary widely.

4. MoE models' KV cache is more complex.

LMCache claims a 10x performance improvement for MoE inference, but KV cache management for MoE models is much more complex than for dense models—different expert's KV caches need to be managed separately, and cache hit rate calculations are more complex. This part is still evolving rapidly.

5. Deployment complexity is not low.

LMCache's MP mode (recommended deployment) requires running lmcache server independently as a sidecar, configuring storage backend, transport layer, eviction policy, compression parameters. For small teams, the operational overhead is not trivial. pip install is just one command, but production deployment is another matter.

Who Should Use It, Who Should Not

Should Use:

• Scenarios with high repetition in long contexts such as multi-turn conversations and Agent workflows
• RAG applications where retrieved documents are heavily repeated across requests
• Large-scale vLLM clusters that need to share KV caches across multiple nodes
• Online services where TTFT is a core SLA metric
• Teams that already have Redis/S3 infrastructure they can reuse

Should Not Use:

• Simple inference scenarios with single requests and no context reuse
• Small teams without distributed storage infrastructure (consider CPU offload mode first)
• Scenarios with extreme quality requirements that cannot tolerate any quality loss
• Teams using engines other than vLLM/SGLang and unwilling to migrate

KV Cache Is Becoming the Operating System of LLM Inference

LMCache's ambition goes beyond being just a caching tool.

One sentence from their blog: "KV cache was not discussed as a feature of an inference engine."—KV cache should not be regarded as a feature of an inference engine, but rather an independent infrastructure layer.

Just as the operating system abstracts memory management from applications, LMCache abstracts KV cache management from inference engines. The inference engine is only responsible for computation; LMCache handles storage, transmission, compression, search, and observability.

This positioning defines its value: Not to make a single request faster, but to make the entire cluster more efficient.

30+ companies use it in production, the PyTorch ecosystem officially embraces it, NVIDIA Dynamo integrates it, vLLM official documentation recommends it—LMCache is evolving from "an open source project" into "a de facto standard."

But the cost of a de facto standard is: you need to invest operational resources for this infrastructure layer. Storage backend, network transmission, cache consistency, eviction policy—each is a classic problem in distributed systems.

LMCache saves GPU computation time, not engineering complexity. Before you decide to install this "operating system," first confirm that your scenario has enough KV cache reuse rate to amortize this complexity.

Related Links:

• GitHub: https://github.com/LMCache/LMCache
• Official Documentation: https://docs.lmcache.ai
• Official Website: https://lmcache.ai
• Technical Report: arXiv 2510.09665
• CacheBlend Paper: arXiv 2405.16444
• Blog: https://blog.lmcache.ai