#serving

Prefill pipelining finally landed. Our 128k-context serving costs should drop meaningfully.

vLLM vs TGI vs llama.cpp server. For a small team serving one 8B model, what's the least ops burden?

Been running a 70B Q4 on one 80GB card for a side project. Key tricks: prefix caching on, max_num_seqs tuned, and a tiny health check for the load balancer. Sharing my config because I wish someone had shared theirs.

We're on vLLM with continuous batching and it's fine, but TGI has better tool-calling support out of the box. Am I missing something? Also considering SGLang. Real-world throughput numbers welcome.