OSDI 2024 Digest

Wed, 10 Jul 2024 00:00:00 +0000

11 papers selected.

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu et al.

TL;DR — Separates the compute-heavy prefill phase from the memory-bound decoding phase onto different GPU pools, eliminating head-of-line blocking and significantly improving LLM serving throughput.

Why notable — Became one of the most influential LLM systems papers of 2024; the prefill–decode disaggregation insight is now widely adopted in production inference stacks (vLLM, SGLang, etc.).

Llm-Serving on Publish Assistant

OSDI 2024 Digest

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving