The Fastest LLM Inference Engine Does Not Exist. xLLM Shows Why.

Chinese AI accelerators are xLLM's starting point, not a secondary port. xLLM is an efficient LLM inference framework designed for enterprise model serving across Ascend, Cambricon, Moore Threads, Hygon, MetaX, and Iluvatar clusters. These accelerators are first-class targets, not an afterthought to a CUDA-first design.

Its economic thesis is simple. Better scheduling, KV reuse, MoE balancing, and hardware-specific operators should produce more SLO-compliant tokens from the accelerators an enterprise already owns. That can raise utilization and lower cost per useful token. The paper measures throughput and latency rather than monetary cost, so the economics still need to be validated in a real deployment.

One result makes the opportunity hard to ignore. The authors report an average 12x throughput improvement over vLLM-Ascend across the tested DeepSeek-R1 configurations on 16 Ascend 910B accelerators. The result is narrow, not universal. The more durable story is the architecture behind it: cluster-level scheduling, shared cache state, and hardware-specific execution that adapts as prefill and decode demand changes.

Five concepts explain most LLM inference performance

An inference engine turns model weights into a service. It allocates memory, batches requests, executes operations, and returns tokens. Engines differ in scheduling, memory, kernels, and distributed execution.

Each text-generation request has two main compute phases.

Prefill processes the prompt in parallel and builds the attention state needed to generate. Long prompts increase this work and delay the first response token.

Decode produces the response one token at a time, repeatedly reading model weights and attention state. Long answers and high concurrency make it expensive.

These phases produce two user-facing metrics. Time to first token (TTFT) measures the delay to the first token. Queueing and prefill usually dominate it. Time per output token (TPOT) measures the interval between later tokens. A response can start quickly yet stream slowly, so both matter.

The KV cache stores attention keys and values already calculated for previous tokens. It avoids recomputing the full history for every new token, but can consume much of the accelerator's memory with long contexts and high concurrency.

Continuous batching lets requests enter and leave a batch between generation steps. It improves utilization, but the scheduler must admit new prefill work without delaying active decodes.

The workload decides which optimization matters

A document service with 20,000-token prompts may be prefill-bound. A reasoning model with long answers may be decode-bound. Repeated system prompts may make prefix-cache hits more valuable than a faster matrix multiplication kernel. Multimodal requests add encoding before prefill.

Hardware, topology, parallelism, and quantization change the ranking again. Raw tokens per second is incomplete because a server can admit more work while making users wait longer. Measure how much work finishes inside the service-level objectives, or SLOs.

xLLM separates cluster decisions from device execution

xLLM is an Apache-2.0 framework for language, multimodal, diffusion, and recommendation models. It centers Ascend NPU, Cambricon MLU, Moore Threads MUSA, Hygon DCU, MetaX MACA, and Iluvatar CoreX. A CUDA development path exists, but the overview and model matrix keep domestic accelerators central. v0.10.0 arrived July 1, 2026. OpenAtom announced the donation signing on June 25; the xLLM repository marked the donation official on July 6.

xLLM's main architectural choice is a split between xLLM-Service and xLLM-Engine.

The service layer sees the cluster. It registers instances, stores metadata, receives metrics, routes requests, plans capacity changes, and manages shared KV information. The engine layer schedules local batches, manages device memory, runs operators, and communicates with other engine processes when the model spans devices.

The technical report describes one device per engine process. Remote procedure calls, or RPCs, coordinate control work. Collectives such as all-reduce and all-to-all move tensors for distributed model execution.

This enables cluster-wide decisions, but xLLM must be evaluated with ETCD, cache transport, routing, and failure behavior, not only as a kernel runtime.

Dynamic PD and multimodal EPD move work between specialized pools

Prefill and decode compete when they share devices. A long prefill can delay decode and push TPOT above its target. Static prefill-decode disaggregation assigns separate device pools to each phase, but a fixed split wastes capacity when traffic changes.

xLLM's disaggregated PD design uses xLLM-Service, ETCD, and separate prefill and decode instances. The report describes dynamic role changes based on TTFT, TPOT, request lengths, and memory use. As demand changes, the planner can shift eligible capacity between the two pools.

Disaggregation adds KV-transfer latency, network load, and another failure path. The right ratio depends on request lengths, arrival bursts, devices, and the protected SLO.

Multimodal inference adds an encode phase for images or video. xLLM's report describes encode-prefill-decode (EPD) profiles such as E+P+D or EP+D. A cluster can isolate a heavy vision encoder when that reduces interference, but keep phases together when transfer overhead costs more than it saves.

Global KV cache and xTensor attack different memory problems

xLLM uses ETCD for cluster registration, load information, and cache metadata. Its Global KV Cache documentation describes routing requests toward instances with reusable prefixes, then offloading and prefetching cache data through local and external storage. The repository credits Mooncake as the basis for this hybrid cache management.

There is an important current limitation: the same documentation says Global KV Cache does not support PD separation and instructs users to set --enable_disagg_pd=false. Features appearing in one architecture do not necessarily compose in the current release.

Within an engine, xTensor addresses another problem. Paged KV caches allocate physical memory in blocks, avoiding maximum-length reservations for every request, but operators need block tables to find those pages. xTensor maps a preallocated physical page pool into contiguous virtual addresses. Operators see a contiguous tensor while physical pages are mapped on demand, reused, and prepared asynchronously.

Like operating-system virtual memory, this preserves paging efficiency but requires accelerator virtual-memory support and backend-specific operator work.

Graph execution saves launch overhead but dynamic shapes complicate memory

LLM execution launches many small operations. Graph execution captures a sequence and replays it, reducing host overhead and accelerator idle gaps.

Serving traffic does not keep one shape. Batch size, prompt length, cache length, and communication volume change continuously. xLLM's Graph Mode design uses shape buckets, dynamic parameters, and Piecewise Graphs. Stable segments are captured while dynamic work can remain in eager execution.

Captured graphs require stable virtual addresses. Separate physical buffers make memory grow toward sum(shape). xLLM gives graph shapes distinct virtual views over shared physical memory, aiming for physical use closer to max(shape).

Two more tools target idle time. Multi-stream parallelism lets one micro-batch compute while another communicates. Expert Parallel Load Balancing monitors mixture-of-experts demand and updates redundant expert placement layer by layer. Their value depends on the model, topology, and traffic.

xLLM, vLLM, SGLang, and TensorRT-LLM optimize different defaults

The four frameworks overlap substantially. The useful comparison is the environment each makes easiest to optimize.

vLLM is a broad baseline with many models, APIs, quantization paths, and hardware plugins. Its V1 scheduler allocates a token budget per iteration. Disaggregated prefill tunes TTFT and inter-token latency independently, but does not automatically improve throughput.

SGLang targets shared-prefix workloads with RadixAttention, plus paged attention, structured outputs, speculative decoding, parallelism, and post-training integrations.

TensorRT-LLM is NVIDIA-specific, combining in-flight batching, paged KV, overlap scheduling, quantization, and disaggregated serving, subject to support matrices.

Hugging Face's Text Generation Inference is in maintenance mode; its maintainers recommend vLLM, SGLang, and local engines for new work.

xLLM's benchmark numbers are strong and narrowly scoped

The xLLM paper compares against vLLM-Ascend 0.10.rc1 and MindIE 2.1.rc1 on Ascend 910B and 910C. These are author-reported results, not an independent cross-hardware leaderboard.

For Qwen-series ShareGPT workloads with 2,048-token inputs and outputs under a 50-millisecond TPOT constraint, the paper reports up to 1.9x vLLM-Ascend throughput on 910B and 2.2x on 910C.

For DeepSeek-R1, it reports an average 12x improvement over vLLM-Ascend on the tested 910B configuration. The 910C baseline is excluded because it missed the required TPOT threshold, so there is no 910C ratio against it.

In a PD comparison with 2,048-token inputs and outputs under a 100-millisecond TPOT constraint, xLLM reaches 11,351.58 tokens per second versus MindIE's 8,476.44, about 34% higher.

The paper does not benchmark SGLang, TensorRT-LLM, or upstream vLLM on NVIDIA. Calling this simply “xLLM versus vLLM” would erase the backend, versions, hardware, workload, and TPOT constraint that produced the result.

Choose by SLO-bounded goodput, not the largest throughput bar

First confirm support for the checkpoint, tokenizer, precision, context length, output behavior, and parallel layout. Compare deterministic outputs before speed.

Then hold the experiment constant:

1. Pin model, framework, backend, driver, firmware, and container versions.
2. Match precision, quantization, and sampling.
3. Fix accelerator count, interconnect, and parallel topology.
4. Replay one request distribution, separating cold and warm caches.
5. Sweep through saturation; report latency percentiles, errors, cache hits, KV transfers, and memory failures.

The final metric should be goodput: requests or tokens completed per second while meeting every required SLO. Work that misses latency or error targets should not make the system appear more productive.

An illustrative decision record would read: “On eight Ascend 910B devices serving checkpoint X in BF16, xLLM v0.10.0 completed [measured percentage] more requests within our p95 TTFT and p99 TPOT targets than vLLM-Ascend version Y under trace Z.”

That result can guide capacity planning. “Framework A produced more tokens per second” cannot.

xLLM is compelling when its hardware focus and cluster architecture match your constraints. vLLM may be the broad default, SGLang may fit prefix-heavy agent traffic, and TensorRT-LLM may suit an NVIDIA-specific deployment. The fastest engine is the one that meets your workload's SLOs on the hardware you operate.

References

1. xLLM repository
2. xLLM Technical Report
3. xLLM feature overview
4. xLLM-Service overview
5. xLLM Global KV Cache
6. xLLM Graph Mode design
7. vLLM documentation
8. SGLang documentation
9. TensorRT-LLM documentation
10. Text Generation Inference maintenance notice