Cloud-Computing on Publish Assistant

SoCC 2024 Digest

Fri, 01 Nov 2024 00:00:00 +0000

12 papers selected.

Queue Management for SLO-Oriented Large Language Model Serving

Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu et al.

TL;DR — A queue management framework that enforces latency SLOs for LLM serving by dynamically routing and prioritizing requests across heterogeneous inference capacity.

Why notable — As LLM deployments move into production clouds, meeting strict time-to-first-token and total latency SLOs becomes critical; this work directly addresses that gap with a practical, deployable solution. It is one of the first papers to treat LLM serving as a cloud SLO-management problem rather than a pure model-optimization problem.

CCGrid 2024 Digest

Mon, 06 May 2024 00:00:00 +0000

10 papers selected.

Fair, Efficient Multi-Resource Scheduling for Stateless Serverless Functions with Anubis

Amit Samanta 0001, Ryan Stutsman

TL;DR — Anubis introduces a fair, multi-resource scheduler for stateless serverless functions that achieves efficiency without sacrificing isolation between tenants.

Why notable — Fairness in serverless resource allocation is an open problem as functions compete for heterogeneous resources (CPU, memory, I/O); Anubis provides a concrete, deployable answer. The work directly addresses a gap in production FaaS platforms where existing schedulers optimize for throughput but ignore per-tenant equity.