The Disaggregated Frontier: Inside the vLLM V1 Rewrite and SGLang EPD Architectures

For years, the engineering playbook for scaling Large Language Model (LLM) serving was deceptively simple: provision massive clusters of NVIDIA H100s, partition model weights across Tensor Parallel (TP) boundaries, and manage incoming requests using simple first-in, first-out schedulers. However, as production workloads shift from brief single-turn prompts to long-context multi-turn reasoning loops and multimodal inputs, this brute-force approach has reached its physical limits. The core bottleneck of modern LLM serving is no longer raw floating-point performance (FLOPS); it is memory bandwidth saturation, Key-Value (KV) cache bloat, and the scheduling interference between the prompt-ingest (prefill) and token-generation (decode) phases.

To bypass these bottlenecks, the open-source LLM infrastructure ecosystem underwent a massive paradigm shift. Powered by deep architectural rewrites—specifically the release of the vLLM V1 engine and SGLang's pioneering of Encoder-Prefill-Decode (EPD) disaggregation—engineers can now build highly decoupled, hardware-efficient serving pipelines. Rather than forcing heterogeneous compute workloads into single monolithic containers, modern infrastructure partitions physical hardware into dedicated pools optimized for specific mathematical phases. This deep dive breaks down the engineering breakthroughs enabling this transition, contrasts the architectural shifts, and provides a direct, production-grade deployment pattern for building a sovereign disaggregated serving cluster.

Technical Benchmarks: Comparing Next-Gen Inference Engines

The primary challenge of LLM serving is that the prefill phase is compute-bound (requiring high FLOPS to ingest prompts), while the decode phase is memory-bandwidth-bound (requiring continuous transfers of model weights and KV caches from high-bandwidth memory to GPU tensor cores).

The table below contrasts how legacy engines handle these phases versus the unified execution paths and physical separations engineered in modern stacks:

1. Inside the vLLM V1 Engine Rewrite: Pure PyTorch and Native Compilation

For over two years, vLLM operated on a legacy execution engine designed around Python-driven model runners. While highly extensible, this architecture suffered from significant Python execution overhead. Every forward pass required traversing deep Python call stacks, calculating metadata on-the-fly, and dynamically allocating memory blocks. Under high-concurrency or sub-millisecond latency targets, this CPU-side bottleneck restricted overall throughput.

The introduction of the vLLM V1 engine represents a total structural rewrite of the execution layer. Rather than treating model compilation as an afterthought, the V1 engine (and its evolution into Model Runner V2) is designed native-first around torch.compile:

1. Static Graph Compilation: The V1 engine leverages PyTorch's native compiler to compile the entire model forward pass into static CUDA graphs. This means that instead of launching thousands of tiny CUDA kernels sequentially from Python, the engine compiles the model into a single, highly-optimized execution plan, dramatically reducing CPU launch overhead.

2. The Zero-Overhead Scheduler: In legacy vLLM, the scheduler worked on the critical path of the GPU execution loop—determining the block layout, grouping requests, and creating metadata while the GPU waited. V1 introduces a decoupled scheduling layer. The scheduler prepares the metadata and constructs the logical batch matrices on the CPU *in parallel* with the GPU running the previous forward pass. The physical transfer of metadata is completely pipelined.

3. Optimized Memory Buffering: By enforcing static shapes and static graph boundaries, the V1 engine drastically reduces memory allocation jitter. The engine reserves a static block of virtual memory at startup, bypassing CUDA's dynamic memory allocator (cudaMalloc) entirely during active execution.

2. The Mechanics of Prefill-Decode (PD) Disaggregation

Even with compiled static graphs, co-located LLM serving suffers from a fundamental conflict: the prefill phase starves the decode phase.

When a client submits a 4,000-token prompt, the prefill engine processes this request in a highly parallel, compute-heavy pass. If this prefill execution is scheduled on the same GPU that is currently running the decode step (generating tokens one-by-one) for fifty other active users, the decode steps must wait. Because the prefill forward pass takes significantly longer than a single decode step, the active users experience a devastating spike in Inter-Token Latency (ITL)—often called "scheduling jitter."

 1
┌─────────────────────────┐
 2
                    │    Inference Router     │
 3
                    └────────────┬────────────┘
 4
                                 │
 5
                 ┌───────────────┴───────────────┐
 6
                 ▼ (Raw Prompts)                 ▼ (Metadata Only)
 7
       ┌───────────────────┐           ┌───────────────────┐
 8
       │   Prefill Pool    │           │    Decode Pool    │
 9
       │   (Compute-Bound) │           │ (Bandwidth-Bound) │
10
       └─────────┬─────────┘           └─────────▲─────────┘
11
                 │                               │
12
                 └───────[ KV Cache Transfer ]───┘
13
                         (RDMA / NIXL over RoCE)

The operational mechanics of this disaggregated architecture are structured in three distinct, sequential steps:

A. The Prefill Instance (The Producer)

The incoming request is routed to a high-compute instance (e.g., NVIDIA H100 or B200) optimized for matrix multiplication. The prefill node processes the entire prompt in a single pass, computing the initial key and value embeddings (the KV cache).

B. High-Speed KV Cache Transfer

Once the prefill phase is complete, the resulting KV cache must be transferred to the Decode Instance. If this transfer occurs over standard 10 GbE TCP/IP networks, the serialization and network latency will completely erase any performance gains. To bypass this, disaggregated engines utilize specialized connectors:

* NixlConnector (NIXL): Leveraging NVIDIA's Inference Xfer Library, this connector implements peer-to-peer memory transfers directly over high-speed InfiniBand or RoCE v2 (RDMA over Converged Ethernet). The KV cache is written directly from the VRAM of the prefill GPU to the VRAM of the decode GPU with zero intermediate CPU copying.

* P2pNcclConnector: For multi-GPU single-node configurations, standard NCCL collective operations are used to stream KV tensors over high-speed NVLink channels.

C. The Decode Instance (The Consumer)

The decode node receives the KV cache directly into its preallocated memory pool. Because the decode node runs *only* sequential decoding passes and is never interrupted by massive prompt-ingest passes, it maintains an incredibly stable, low-latency token generation rate.

3. SGLang and the Breakthrough of Encoder-Prefill-Decode (EPD) Disaggregation

As Vision-Language Models (VLMs) like Qwen-2.5-VL and LLaVA gain enterprise prominence, standard Prefill-Decode disaggregation breaks down. The bottleneck shifts to the processing of high-resolution image and video inputs.

In a standard VLM forward pass, a single image might be converted into 1,024 or even 4,096 visual tokens by a Vision Transformer (ViT) encoder. Running these heavy visual encoders on standard text-serving pipelines introduces severe processing delays. SGLang (v0.5+) solves this by introducing a three-tier architecture: Encoder-Prefill-Decode (EPD) Disaggregation.

 1
[User Request (Image + Prompt)]
 2
               │
 3
               ▼
 4
┌──────────────────────────────┐
 5
│  Tier 1: Vision Encoders     │  <-- Highly parallel, non-autoregressive ViT execution
 6
└──────────────┬───────────────┘
 7
               │ (Visual Token Embeddings)
 8
               ▼
 9
┌──────────────────────────────┐
10
│   Tier 2: Language Prefill   │  <-- Prepares text prompt + image embeddings 
11
└──────────────┬───────────────┘
12
               │ (Logical KV Cache Blocks)
13
               ▼
14
┌──────────────────────────────┐
15
│   Tier 3: Language Decode    │  <-- Sequential, ultra-fast autoregressive generation
16
└──────────────────────────────┘

By decoupling the vision encoding, SGLang ensures that the massive compute requirements of visual preprocessing do not block the active generation phases of the downstream language model. The visual tokens are computed on dedicated visual encoding nodes, serialized, and streamed to the language prefill nodes, which then construct the final text-visual KV blocks for the decode engines.

4. Execution Walkthrough: Deploying a Disaggregated vLLM Cluster

This deployment pattern demonstrates how to configure and launch a disaggregated prefill-decode pipeline using upstream vLLM (v0.8.0+) with native P2pNcclConnector communications. This setup assumes two distinct GPU nodes (or two isolated GPU groups on a single node) sharing a high-speed network path.

Step 1: Initialize the Prefill (KV Producer) Node

On the compute-heavy node designated as the KV cache producer, launch the vLLM server. We configure the --kv-transfer-config flag with a JSON block defining its role as the producer.

 1
# Start the Prefill Server on Port 8100
 2
vllm serve Qwen/Qwen2.5-32B-Instruct \
 3
  --port 8100 \
 4
  --trust-remote-code \
 5
  --gpu-memory-utilization 0.90 \
 6
  --kv-transfer-config '{
 7
    "kv_connector": "P2pNcclConnector",
 8
    "kv_role": "kv_producer",
 9
    "kv_rank": 0,
10
    "kv_parallel_size": 2,
11
    "kv_ip": "10.0.1.10",
12
    "kv_port": "14579",
13
    "kv_buffer_size": "2e9"
14
  }'

Step 2: Initialize the Decode (KV Consumer) Node

On the memory-bandwidth-optimized node designated as the KV cache consumer, launch the corresponding vLLM server. Ensure the model, tokenizer, and network configurations match the producer exactly.

 1
# Start the Decode Server on Port 8200
 2
vllm serve Qwen/Qwen2.5-32B-Instruct \
 3
  --port 8200 \
 4
  --trust-remote-code \
 5
  --gpu-memory-utilization 0.90 \
 6
  --kv-transfer-config '{
 7
    "kv_connector": "P2pNcclConnector",
 8
    "kv_role": "kv_consumer",
 9
    "kv_rank": 1,
10
    "kv_parallel_size": 2,
11
    "kv_ip": "10.0.1.10",
12
    "kv_port": "14579",
13
    "kv_buffer_size": "2e9"
14
  }'

Step 3: Implement an Intelligent Orchestration Routing Script

Because the prefill and decode instances run as independent servers, we need a lightweight orchestration layer (or API router) to coordinate sending the initial prompt to the prefill instance, extracting the generated metadata, and streaming the generation task to the decode node.

The script below demonstrates this routing mechanism using a standard Python client workflow:

 1
import json
 2
import requests
 3
 4
PREFILL_ENDPOINT = "http://10.0.1.10:8100/v1/completions"
 5
DECODE_ENDPOINT = "http://10.0.1.11:8200/v1/completions"
 6
 7
def execute_disaggregated_request(prompt: str):
 8
    # 1. Package the request with specific routing indicators
 9
    payload = {
10
        "model": "Qwen/Qwen2.5-32B-Instruct",
11
        "prompt": prompt,
12
        "max_tokens": 256,
13
        "temperature": 0.2,
14
        # Indicate that the prefill engine should process and immediately export the KV Cache
15
        "extra_body": {
16
            "kv_transfer_action": "export"
17
        }
18
    }
19
    
20
    # 2. Trigger the prefill phase
21
    print("[Routing] Initiating prefill phase on Producer Node...")
22
    prefill_response = requests.post(PREFILL_ENDPOINT, json=payload)
23
    prefill_data = prefill_response.json()
24
    
25
    # 3. Extract transfer tokens/metadata mapping the cached keys
26
    transfer_token = prefill_data.get("kv_transfer_token")
27
    print(f"[Routing] Prefill completed. KV Cache Token: {transfer_token}")
28
    
29
    # 4. Hand off token generation directly to the Decode Node
30
    decode_payload = {
31
        "model": "Qwen/Qwen2.5-32B-Instruct",
32
        "prompt": prompt,
33
        "max_tokens": 256,
34
        "temperature": 0.2,
35
        "extra_body": {
36
            "kv_transfer_action": "import",
37
            "kv_transfer_token": transfer_token
38
        }
39
    }
40
    
41
    print("[Routing] Handing off task to Consumer Node for streaming generation...")
42
    decode_response = requests.post(DECODE_ENDPOINT, json=decode_payload, stream=True)
43
    
44
    for line in decode_response.iter_lines():
45
        if line:
46
            print(line.decode('utf-8'))
47
48
if __name__ == "__main__":
49
    test_prompt = "Explain the difference between CUDA graphs and eager execution in PyTorch with detailed assembly-level differences."
50
    execute_disaggregated_request(test_prompt)

5. The Reality Check: Production Gotchas and Hardware Realities

While the theoretical benchmarks and throughput improvements of disaggregated architectures are incredibly compelling, deploying these systems in production environments reveals a series of severe real-world "gotchas" that documentation rarely covers:

* Network Jitter Erases Performance Gains: Prefill-Decode disaggregation is entirely dependent on network speed. If your cluster is not connected via dedicated physical high-speed interconnects—specifically InfiniBand or RoCE v2 with line speeds of at least 100 Gbps to 400 Gbps—the latency of serializing and transmitting the KV cache across nodes will actually exceed the latency of running the prefill phase locally. If you try to run disaggregation over standard Gigabit Ethernet or standard cloud VPC virtual networks, your latency performance will drastically degrade.

* The Cold-Start Compilation Freeze: By moving to a torch.compile and static CUDA graph architecture, vLLM V1 models must be compiled on startup. For massive models (e.g., 70B parameters or Mixture-of-Experts architectures), this initial JIT (Just-In-Time) compilation can freeze the serving container for up to 12 minutes upon boot. If your Kubernetes cluster is configured with default liveness or readiness probes that do not account for this massive startup delay, your orchestrator will repeatedly kill the pod mid-compilation, locking your deployment into an infinite crash loop.

* Dynamic Outlier Activation Crashing FP8: Modern disaggregated stacks highly utilize FP8 precision to maximize KV cache compression and memory throughput. However, models undergoing heavy reasoning chains (such as DeepSeek-R1 or Qwen-2.5-Math) occasionally produce extreme activation outliers during long context decodes. These mathematical outliers can cause catastrophic precision degradation in FP8 mixed-precision kernels, culminating in sudden NaN outputs or causing the underlying CUDA kernels to throw unrecoverable memory segment faults, instantly crashing the serving pod.

Verdict: The Sovereign Case for Disaggregated Architectures

For infrastructure engineers, CTOs, and sovereign platform builders, the maturation of vLLM V1 and SGLang's EPD disaggregation completely redefines the economics of AI scale.

The traditional path of scaling LLM platforms required purchasing massive, homogeneous clusters of ultra-high-end GPUs, keeping them locked under monolithic, unified workloads. Under a disaggregated paradigm, you can optimize your hardware allocation on a granular, mathematical level:

* Deploy highly parallel, high-power compute clusters (such as H100s or B200s) exclusively as Prefill Pools to process prompts instantly.

* Deploy highly cost-efficient, memory-dense hardware (such as L40S, A100s, or AMD Instinct MI300X clusters) as Decode Pools to stream tokens.

This is the ultimate evolution of sovereign AI engineering: minimizing waste, maximizing physical hardware utility, and maintaining total control over latency, infrastructure costs, and data boundaries. The era of brute-force compute allocation is officially over; the era of granular, disaggregated systems architecture has arrived.