The Rise of Prefill-Decode Disaggregation: Scaling Sovereign LLM Serving Beyond Collocation

The architecture of modern Large Language Model (LLM) serving has hit a fundamental scaling bottleneck. In traditional collocated inference, a single GPU or a tightly coupled cluster processes both the initial prompt analysis (prefill) and the subsequent token generation (decode) on the same compute engines. While conceptually simple, this approach forces two diametrically opposed computational workloads to share the same physical silicon, leading to severe resource scheduling conflicts, extreme latency spikes, and degraded throughput under high concurrency.

To break this bottleneck, state-of-the-art serving frameworks like vLLM and SGLang have transitioned to Prefill-Decode (PD) Disaggregation. By physically separating the compute-bound prefill phase from the memory-bandwidth-bound decode phase onto distinct pools of GPUs, disaggregated serving eliminates head-of-line blocking and allows infrastructure architects to optimize hardware specifically for each computational profile.

This deep-tech guide details the mechanics of PD disaggregation, calculates the mathematical constraints of Key-Value (KV) cache transfer payloads, analyzes the implementation differences between vLLM and SGLang, and provides a production-ready deployment blueprint for architecting a sovereign, disaggregated LLM serving cluster.

Technical Benchmarks: Collocated Serving vs. PD Disaggregation

Before diving into the network and physical memory layers, we must evaluate the core performance differences between standard collocated batching and disaggregated serving configurations.

The Collocation Bottleneck: Why Standard Serving Stalls

An LLM inference query has two distinct physical phases, each bound by completely different physical limits:

1. The Prefill Phase (Compute-Bound): When a user sends a prompt, the engine processes the entire sequence in parallel. This is highly parallelizable, utilizing matrix multiplication kernels that saturate the GPU's tensor cores. This phase is heavily compute-bound and scales with the raw FLOP performance of the hardware.

2. The Decode Phase (Memory-Bandwidth Bound): Once the first token is generated, the model enters autoregressive generation, reading and writing single tokens sequentially. Because only one token is evaluated at a time, tensor cores sit idle while the entire model weight matrix and the accumulated KV cache must be fetched from High-Bandwidth Memory (HBM) to High-Speed SRAM for *every single token*. This phase is heavily memory-bandwidth bound.

Head-of-Line Blocking Visualized

When these two workloads are collocated on the same hardware, incoming prefill requests interrupt running decode loops. Since a prefill pass over a large context requires significantly more compute time than a single decode step, active decodes must halt. This phenomenon, known as Head-of-Line (HoL) Blocking, causes massive spikes in Inter-Token Latency (ITL), causing text generation to stutter for active users.

 1
COLLOCATED INFERENCE:
 2
Time  ──►
 3
[GPU 0] ┌───────────┬───┬───┬───────────┬───┬───┬───┐
 4
        │ Prefill   │ D1│ D1│ Prefill   │ D1│ D2│ D1│  <-- Prefills block active decodes (D1, D2)
 5
        │ (Query 1) │   │   │ (Query 2) │   │   │   │      causing severe latency spikes.
 6
        └───────────┴───┴───┴───────────┴───┴───┴───┘
 7
 8
DISAGGREGATED INFERENCE:
 9
[Prefill Pool] ┌───────────────────────┬───────────────────────┐
10
               │ Prefill Query 1       │ Prefill Query 2       │  <-- Raw compute-bound hardware
11
               └───────────────────────┴───────────────────────┘
12
                                   │ (KV-Cache Transfer over NVLink/RoCE)
13
                                   ▼
14
[Decode Pool]  ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┬───┐
15
               │ D1│ D1│ D1│ D1│ D1│ D2│ D1│ D2│ D1│ D2│ D1│ D2│  <-- Flat, jitter-free decode loops
16
               └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘

Disaggregation physically decouples these timelines. Prefill instances ingest requests, compute the initial KV cache, and stream the generated cache matrices over high-speed networks to dedicated decode instances, which continuously append to their local pages without experiencing compute-induced jitter.

The Mathematics of KV-Cache Transfers: The Payload Challenge

Moving the Key-Value (KV) cache between different physical server nodes introduces a critical networking bottleneck: the transfer time must be significantly lower than the prefill compute time, or the network overhead will completely negate the advantages of disaggregation.

Let's compute the physical payload size of a KV cache transfer.

1. Traditional Multi-Head Attention (MHA) Payload

For traditional Multi-Head Attention, the total memory footprint of the KV cache for a sequence scales linearly with the sequence length, the number of layers, head count, and precision bytes:

$$\text{Size}_{\text{MHA}} = 2 \times L \times N_{\text{layers}} \times H_{\text{kv}} \times D_{\text{head}} \times B$$

Where:

* $L$ = Sequence Length (Tokens)

* $N_{\text{layers}}$ = Number of Transformer Layers

* $H_{\text{kv}}$ = Number of Key-Value Heads (equal to query heads in MHA; reduced in GQA)

* $D_{\text{head}}$ = Hidden Dimension size per Head

* $B$ = Bytes per Precision Float (e.g., 2 bytes for FP16/BF16, 1 byte for quantized FP8)

For a 32,768 (32K) token prompt context:

$$\text{Size}_{\text{GQA}} = 2 \times 32,768 \times 80 \times 8 \times 128 \times 2 = 10,737,418,240 \text{ bytes} \approx 10.74 \text{ GB}$$

Over a standard 100 Gbps (Gigabits per second) network, the theoretical minimum transfer time for this single cache is:

$$\text{Transfer Time}_{\text{100G}} = \frac{10.74 \text{ GB} \times 8}{100 \text{ Gbps}} \approx 0.86 \text{ seconds (860 ms)}$$

An 860 ms network transfer latency is completely unacceptable for real-time serving, as it exceeds typical prefill compute times on modern GPUs.

2. Multi-head Latent Attention (MLA) Payload

The math shifts dramatically when serving architectures utilize Multi-head Latent Attention (such as DeepSeek-V3). MLA compresses the key-value states into a low-rank joint latent space, massively shrinking the network payload size:

$$\text{Size}_{\text{MLA}} = L \times N_{\text{layers}} \times (d_c + d'_R) \times B$$

Where:

* $d_c$ = Compressed Key-Value Latent Dimension (typically 512)

* $d'_R$ = Decoupled Key Rotary Position Embedding (RoPE) Dimension (typically 64)

For DeepSeek-V3 ($N_{\text{layers}} = 61$, served with FP8 quantization where $B = 1$ byte) at the same 32K token sequence length:

$$\text{Size}_{\text{MLA}} = 32,768 \times 61 \times (512 + 64) \times 1 = 1,151,336,448 \text{ bytes} \approx 1.15 \text{ GB}$$

This represents a 89.3% reduction in network payload size compared to Grouped-Query Attention. The theoretical transfer time over the same 100 Gbps network drops to:

$$\text{Transfer Time}_{\text{100G}} = \frac{1.15 \text{ GB} \times 8}{100 \text{ Gbps}} \approx 0.092 \text{ seconds (92 ms)}$$

At 92 ms, the network transfer overhead falls well within acceptable operational boundaries, enabling seamless distributed disaggregated serving of frontier-scale architectures.

Architectural Deep Dive: vLLM vs. SGLang on PD Disaggregation

Production engines have implemented distinct strategies to address the orchestration and transfer of these massive memory states.

vLLM’s Implementation: The kv_transfer & MoRI-IO Pipeline

vLLM integrates disaggregated serving natively within its unified engine via the vllm/distributed/kv_transfer API.

* Modular RDMA Interface for IO (MoRI-IO): To minimize inter-node transfer times, vLLM relies on MoRI-IO, an open-source, high-performance point-to-point communication engine designed for GPUDirect RDMA (GDR) and AMD xGMI. MoRI-IO establishes multi-queue pair parallel transfers directly between GPU memory spaces, bypassing CPU-host memory entirely.

* Active Push vs. Lazy Pull Modes: vLLM supports both Read (Lazy Pull) and Write (Active Push) modes. In Write Mode, the prefill runner actively streams the KV blocks into pre-allocated memory structures on the target decode node while finishing its computation. In Read Mode, the decode node fetches only the required block indexes from a unified shared-buffer database upon query request.

* Single-Node Disaggregation: For sovereign deployments running on unified systems (like an 8x H100 system), vLLM allows partitioning the local system. By dedicating GPUs 0-3 for prefill and GPUs 4-7 for decode, vLLM avoids network socket bottlenecks, performing memory copies directly via ultra-fast intra-chassis NVLink (up to 900 GB/s bidirectional throughput), dropping transfer latency to less than 5ms.

SGLang’s Implementation: Rust Routing and the Bootstrap Protocol

SGLang optimizes disaggregation with a focus on web-scale multi-user workloads using a decentralized, state-aware architectural model.

 1
┌────────────────────────────────────────┐
 2
              │             sgl-router                 │
 3
              │  (State-Aware Load Balancer / Proxy)   │
 4
              └────────────────────────────────────────┘
 5
                   /                              \
 6
 (Dispatch Prefill)                              (Dispatch Decode)
 7
                 /                                  \
 8
                ▼                                    ▼
 9
       ┌────────────────┐                    ┌────────────────┐
10
       │ Prefill Worker │                    │ Decode Worker  │
11
       │ (Bootstrap 8998)│◄─ ─ ─ ─ ─ ─ ─ ─ ─ ─►│ (Bootstrap 8998)│
12
       └────────────────┘  Direct RDMA/NVLink└────────────────┘
13
                          KV-Cache Transfer

* The sgl-router Gateway: SGLang relies on a high-performance Rust-based routing proxy (sgl-router). Unlike basic round-robin proxies, the router monitors the exact memory map of both prefill and decode instances. When a request arrives, it pairs a prefill node and a decode node based on prefix-cache state and dispatches the task with a synchronized rendezvous token.

* The Bootstrap Handshake: SGLang deploys a lightweight bootstrap process (typically on port 8998) on every worker instance. The paired prefill and decode instances use this daemon to execute a handshake, bypass the routing proxy entirely, and spin up a direct peer-to-peer RDMA or NVLink channel for the KV-cache transfer.

* Mismatched Tensor Parallelism (TP): SGLang natively supports asymmetric TP configurations. For instance, you can configure the Prefill pool to run at TP=4 (allocating 4 GPUs to solve a single prefill rapidly to minimize TTFT) while the Decode pool runs at TP=1 (dedicating 1 GPU per decode task to maximize concurrent generation density). The transfer engine automatically maps and replicates the attention keys across mismatched GPU matrices during the network transfer.

Deploying a Disaggregated Serving Stack with vLLM

We will now deploy a disaggregated serving stack on a single-node system containing 8x NVIDIA H100 (80GB VRAM) GPUs using vLLM.

To maximize throughput, we will partition the 8 GPUs:

* GPUs 0-3 (Prefill Node): Configured as a high-compute prefill instance running Tensor Parallelism 4 (TP=4).

* GPUs 4-7 (Decode Node): Configured as a high-density decode instance running Tensor Parallelism 4 (TP=4) in listener mode.

Step 1: Install vLLM with RDMA & MoRI-IO Support

Ensure your system has CUDA 12.1+ and UCX (Unified Communication X) configured for GPUDirect RDMA.

 1
# Update local packages
 2
sudo apt-get update && sudo apt-get install -y libucx-dev
 3
 4
# Create isolated Python environment
 5
python3 -m venv vllm-disaggregated
 6
source vllm-disaggregated/bin/activate
 7
 8
# Install vLLM with CUDA and the optimized KV transfer dependencies
 9
pip install --upgrade pip
10
pip install vllm>=0.6.0

Step 2: Configure and Launch the Decode Server (Receiver)

The decode instance must launch first and listen for incoming KV cache streams. We bind it to GPUs 4-7 and configure the kv-transfer module to operate in receiver mode on port 50010.

Create a file named launch_decode.sh:

 1
#!/bin/bash
 2
export CUDA_VISIBLE_DEVICES=4,5,6,7
 3
 4
# Launch vLLM in receiver (decode) mode
 5
python3 -m vllm.entrypoints.openai.api_server \
 6
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 7
    --tensor-parallel-size 4 \
 8
    --port 8001 \
 9
    --host 0.0.0.0 \
10
    --gpu-memory-utilization 0.90 \
11
    --kv-transfer-role receiver \
12
    --kv-transfer-ip 127.0.0.1 \
13
    --kv-transfer-port 50010 \
14
    --trust-remote-code

Step 3: Configure and Launch the Prefill Server (Sender)

The prefill instance handles the initial prompt evaluations. We bind it to GPUs 0-3 and configure the kv-transfer module to act as a sender targeting the decode server's transfer port.

Create a file named launch_prefill.sh:

 1
#!/bin/bash
 2
export CUDA_VISIBLE_DEVICES=0,1,2,3
 3
 4
# Launch vLLM in sender (prefill) mode
 5
python3 -m vllm.entrypoints.openai.api_server \
 6
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 7
    --tensor-parallel-size 4 \
 8
    --port 8000 \
 9
    --host 0.0.0.0 \
10
    --gpu-memory-utilization 0.90 \
11
    --kv-transfer-role sender \
12
    --kv-transfer-ip 127.0.0.1 \
13
    --kv-transfer-port 50010 \
14
    --trust-remote-code

Step 4: Implement a Distributed Coordinating Client

With both nodes running, we require a lightweight orchestrator client. The client routes the user's prompt first to the Prefill server to build and transfer the KV cache, and then redirects the generation task to the Decode server to fetch the streaming response.

Create disaggregated_client.py:

 1
import time
 2
import requests
 3
import json
 4
 5
PREFILL_URL = "http://localhost:8000/v1/chat/completions"
 6
DECODE_URL = "http://localhost:8001/v1/chat/completions"
 7
 8
def execute_disaggregated_query(prompt_content: str):
 9
    payload = {
10
        "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
11
        "messages": [
12
            {"role": "user", "content": prompt_content}
13
        ],
14
        "temperature": 0.6,
15
        "max_tokens": 1024,
16
        "stream": True
17
    }
18
    
19
    print("[INFO] Phase 1: Initiating prefill pass on GPUs 0-3...")
20
    start_time = time.time()
21
    
22
    # 1. Warm prefill and trigger the internal KV-cache push
23
    # We use a custom header to inform the orchestrator of the pipeline status
24
    prefill_headers = {"X-Inference-Phase": "prefill-only"}
25
    prefill_response = requests.post(PREFILL_URL, json=payload, headers=prefill_headers, stream=True)
26
    
27
    if prefill_response.status_code != 200:
28
         print(f"[ERROR] Prefill phase failed: {prefill_response.text}")
29
         return
30
         
31
    # Consume prefill response to ensure KV cache completes computation
32
    for line in prefill_response.iter_lines():
33
        pass
34
        
35
    ttft = time.time() - start_time
36
    print(f"[SUCCESS] Prefill phase complete. TTFT: {ttft:.4f}s. KV cache pushed.")
37
    
38
    # 2. Query the decode engine (GPUs 4-7) for generation
39
    print("[INFO] Phase 2: Redirecting generation to Decode pool (GPUs 4-7)...")
40
    decode_headers = {"X-Inference-Phase": "decode-only"}
41
    
42
    decode_response = requests.post(DECODE_URL, json=payload, headers=decode_headers, stream=True)
43
    
44
    print("\n=== SYSTEM RESPONSE (Streaming) ===")
45
    for chunk in decode_response.iter_lines():
46
        if chunk:
47
            decoded_chunk = chunk.decode('utf-8').strip()
48
            if decoded_chunk.startswith("data: "):
49
                data_str = decoded_chunk[6:]
50
                if data_str == "[DONE]":
51
                    break
52
                try:
53
                    data_json = json.loads(data_str)
54
                    token = data_json["choices"][0]["delta"].get("content", "")
55
                    print(token, end="", flush=True)
56
                except json.JSONDecodeError:
57
                    continue
58
    print("\n\n=== GENERATION COMPLETE ===")
59
    print(f"Total Turn Latency: {time.time() - start_time:.4f}s")
60
61
if __name__ == "__main__":
62
    test_prompt = """
63
    Explain the mathematical proof behind multi-query attention optimization 
64
    and detail how it alters the spatial layout of high-bandwidth memory allocations.
65
    Provide your explanation formatted as an in-depth lecture script.
66
    """
67
    execute_disaggregated_query(test_prompt)

The Reality Check: Operational Gotchas of Disaggregated Serving

Despite the massive theoretical throughput and latency gains, implementing PD disaggregation in real-world production environments exposes several critical architectural risks:

* Interconnect Bandwidth Bottlenecks: If your cluster relies on standard 10 GbE or 25 GbE Ethernet interfaces without RoCE v2 or InfiniBand, the network latency required to transmit the KV cache will exceed the local computation time of the prefill. In this state, disaggregated serving degrades performance, making standard collocated batching faster. GPUDirect RDMA is a hard requirement for multi-node deployments.

* The "Short Prompt" Overhead Penalty: For small prompts (e.g., simple instructions under 256 tokens), the prefill compute cost is extremely minor (typically sub-10ms). The overhead of initiating network handshakes, serializing the cache, and transmitting it across sockets can be higher than simply recalculating the prefill directly on the decode node. Production routing layers must implement threshold-based bypass filters to route short prompts to collocated engines.

* Memory Fragmentation in Decode Pools: Decode nodes are constantly ingesting and discarding physical KV cache structures from various network clients. Over time, this dynamic block swapping can trigger severe page fragmentation. While PagedAttention mitigates this locally, the disaggregated interface must synchronize garbage collection routines to prevent nodes from throwing out-of-memory (OOM) exceptions due to fragmented page tables.

The Sovereign Verdict

For teams engineering sovereign artificial intelligence infrastructure, the architectural decision matrix is clear.

If you are running a deployment characterized by highly dynamic conversational systems, high-concurrency multi-user agents, or dense Retrieval-Augmented Generation (RAG) pipelines with large search payloads, Prefill-Decode Disaggregation is a critical upgrade. Its ability to completely decouple prefill-induced compute spikes from sequential generation loops resolves the single greatest cause of user-facing latency jitter.

However, if your primary workload consists of offline batch processing, document summarization, or single-user long-reasoning tasks (such as deep planning chains), collocation remains the most resource-efficient pattern.

By separating compute-heavy prompt ingestion from memory-heavy sequence generation, disaggregated architectures free infrastructure developers from the physical limits of single-GPU memory architectures—establishing a new standard for high-performance, predictable, and scalable sovereign AI operations.