Mooncake
Mooncake is an open-source LLM serving platform designed for efficient distributed inference through a KVCache-centric disaggregated architecture. It enables multiple serving systems (vLLM, SGLang, TensorRT-LLM, LMDeploy) to share KV cache across instances and machines, reducing redundant computation and memory overhead. Originally built to power Kimi (a leading LLM service by Moonshot AI), it is now in active development with broad integration across the ML inference ecosystem.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | kvcache-ai/Mooncake |
| Owner | kvcache-ai |
| Primary language | C++ |
| License | Apache-2.0 — OSI-approved |
| Stars | 5.8k |
| Forks | 937 |
| Open issues | 474 |
| Latest release | v0.3.11.post1 (2026-05-23) |
| Last updated | 2026-07-08 |
| Source | https://github.com/kvcache-ai/Mooncake |
What Mooncake is
Mooncake implements a disaggregated serving layer that decouples prefill and decode phases, using RDMA-based peer-to-peer transfer for zero-copy KV cache movement. It provides both a KVCache Store (distributed cache pool) and a TransferEngine (high-performance inter-node data transfer), supporting CUDA ≤12.9, CUDA 13.0/13.1, and NPU backends. The platform integrates natively with vLLM, SGLang, TensorRT-LLM, and other systems, enabling cross-instance embedding and state reuse at scale.
Get the Mooncake source
Clone the repository and explore it locally.
git clone https://github.com/kvcache-ai/Mooncake.gitcd Mooncake# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- CUDA/NPU toolkit version compatibility: Ensure cluster GPUs match supported CUDA versions (≤12.9, 13.0/13.1) or use non-CUDA wheels. NPU variant requires Ascend/equivalent hardware.
- RDMA infrastructure: Set up and validate RDMA networking (Infiniband or RoCE) across cluster nodes. Mooncake's zero-copy transfer efficiency depends on low-latency interconnect; fall back to TCP if unavailable.
- Integration point selection: Decide whether to use Mooncake Store (distributed KV pool), TransferEngine (data transfer), or both. Each integration point (vLLM connector, SGLang backend, TRT-LLM plugin) has slightly different setup and configuration.
- Memory and scaling planning: Profile KV cache size per model and instance to dimension KVCache Store. Account for cross-instance data transfer overhead in throughput and latency budgeting.
- Operational monitoring: Establish metrics for KV cache hit rates, RDMA bandwidth utilization, and transfer latency. Missing observability may delay troubleshooting in multi-node deployments.
When to avoid it — and what to weigh
- Single-machine, latency-critical inference — Mooncake's disaggregated architecture and RDMA setup add complexity. For simple single-GPU or single-machine deployments with low latency requirements, standard vLLM or TensorRT-LLM without Mooncake overhead is simpler.
- Network-constrained environments — Mooncake relies on efficient RDMA for cross-instance KV cache transfer. Without high-bandwidth, low-latency interconnect (Infiniband, RoCE), or on CPU-only infrastructure, the benefits diminish significantly.
- Proprietary inference frameworks or custom kernels — Mooncake integrates with vLLM, SGLang, TensorRT-LLM, LMDeploy, and similar frameworks. If you require a proprietary or highly customized inference engine, integration effort may exceed benefits.
- Experimental or early-stage production workloads — While actively maintained, Mooncake is still iterating (latest v0.3.11.post1, 474 open issues). Not yet recommended as sole serving platform for mission-critical, zero-downtime production if your team lacks ML infrastructure expertise.
License & commercial use
Mooncake is licensed under Apache License 2.0 (Apache-2.0), a permissive OSI-approved open-source license. The license permits commercial use, modification, and distribution, subject to license reproduction and disclaimer requirements.
Apache-2.0 is a permissive license explicitly allowing commercial use. However, the project is maintained by Moonshot AI and community contributors; no formal commercial support or SLA is stated in provided data. Organizations using Mooncake in production should: (1) review the full LICENSE-APACHE file; (2) confirm compatibility with their own licensing obligations; (3) evaluate community support (GitHub issues, Slack) versus paid support if needed. Requires review of any custom modifications and third-party dependencies.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | High |
| DEV.co fit | Good |
| Assessment confidence | High |
Apache-2.0 license includes liability disclaimers. Code is open-source and subject to community security review. RDMA data transfer should be evaluated for network isolation and access control within your infrastructure (RDMA typically assumes trusted network). No formal security audit or CVE tracking data provided; assess via GitHub advisories and community reports. When handling sensitive data (e.g., user prompts in shared KV cache), ensure cache isolation policies align with data governance requirements.
Alternatives to consider
vLLM (standalone, without Mooncake)
Simple, single-node or basic distributed inference. Avoids Mooncake's multi-instance complexity but loses KV cache sharing and cross-instance SLO benefits. Suitable if your cluster is small or latency margins are loose.
TensorRT-LLM (with native KV cache management)
Proprietary NVIDIA serving stack with tightly optimized kernels. Provides good single-instance throughput and is well-supported. Lacks out-of-the-box multi-instance KV cache sharing; Mooncake now offers TRT-LLM plugins to add this.
SGLang (without Mooncake Store/TransferEngine)
Supports speculative decoding and RadixAttention for cache sharing on single node. For multi-node deployments, does not natively share KV cache across instances without Mooncake integration.
Build on Mooncake with DEV.co software developers
Mooncake can reduce your inference costs and increase throughput through intelligent KV cache sharing. Let's assess your cluster topology and model workload to determine fit and ROI. Contact our team to discuss architecture, RDMA setup, and integration planning.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
Mooncake FAQ
Do I need to rewrite my existing vLLM or SGLang code to use Mooncake?
What is the typical performance gain from using Mooncake?
Can I use Mooncake on CPU-only clusters or without RDMA?
Is Mooncake production-ready?
Software developers & web developers for hire
From first prototype to production, DEV.co delivers software development services around tools like Mooncake. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across ai frameworks and beyond.
Ready to optimize LLM inference at scale?
Mooncake can reduce your inference costs and increase throughput through intelligent KV cache sharing. Let's assess your cluster topology and model workload to determine fit and ROI. Contact our team to discuss architecture, RDMA setup, and integration planning.