How to Cut LLM Cost and Latency: Caching, Batching, Routing

Q: In LLM inference, which stage mainly determines the 'time to first token (TTFT)'?

Prefill (computing the whole input at once, compute-bound)。Prefill computes the input and sets first-token latency; decode emits tokens one by one and sets per-token speed.

Q: True or false: 'When an agent is slow and expensive, the most direct, effective fix is to switch to a bigger, stronger model.'

False。A bigger model is often pricier and slower; first optimize architecture + prompt caching (cache reads at 10% price) / batching (50% off) / routing / compression / speculative decoding. Multi-agent already amplifies cost ~15x, so architecture comes first.

Where latency comes from: prefill sets the first token, decode sets each token

Before optimizing, understand that LLM inference has two stages: prefill (compute the whole input at once, compute-bound, sets the time to first token, TTFT) and decode (emit tokens one by one, memory-bandwidth-bound, sets per-token speed). One key asymmetry: output tokens are pricier and slower than input — so cutting generation length, capping max_tokens, and streaming to improve perceived speed is the highest-ROI first move.

The biggest memory hog is the KV cache (storing already-computed attention): naive management wastes 60–80% of memory; PagedAttention (vLLM) borrows the OS's paging idea to cut waste to under 4% and multiply throughput. Its companion, continuous batching, drops scheduling from 'request-level' to 'iteration-level' — as soon as one sequence finishes, a new request takes its slot — greatly raising throughput.

💡The first move is always 'generate less': output is pricey and slow, so keep it short and stream when you can — perceived speed and the bill improve together.

Five practical moves to cut cost and speed things up

① Prompt caching: a cache-hit input is billed at just 10% of the base input price and lowers time-to-first-token — especially sweet with a large fixed prefix (system prompt / documents). ② Batches API: trade real-time for a 50% discount on input and output tokens, returned within 24 hours — good for offline bulk work. ③ Model cascade / routing (FrugalGPT): use a cheap model first and escalate to an expensive one only when it's not good enough — saving up to 98% while keeping GPT-4-level quality.

④ Prompt compression (LLMLingua): a small model deletes redundant tokens by information content, up to 20x compression with almost no performance drop. ⑤ Speculative decoding: a small draft model guesses a span and the big model verifies in parallel — lossless speedup (~2–3×) (output distribution unchanged, at the cost of running an extra small draft model). The myth to bust most: thinking 'just use a bigger model' solves everything — a multi-agent architecture alone amplifies token cost ~15x, so optimize the architecture and the moves above first, then talk about swapping models.

⚠️Myth: 'if it's not working, switch to a bigger model.' First cut cost with caching / batching / routing / compression / speculative decoding; a bigger model is the last resort, not the first.

自测 · 学完检查一下

想真正动手做题、记进度、攒连胜？到互动课里练。

In LLM inference, which stage mainly determines the 'time to first token (TTFT)'?

答案：Prefill (computing the whole input at once, compute-bound)

Prefill computes the input and sets first-token latency; decode emits tokens one by one and sets per-token speed.

Which is a way to 'cut cost while keeping quality'?

答案：Model cascade / routing: use a cheap model first, escalate to an expensive one only when needed (FrugalGPT saves up to 98%)

FrugalGPT cascades from cheap to expensive by confidence, keeping GPT-4-level quality while saving up to 98%; prompt caching / batches / compression / speculative decoding are also cost/speed moves.

True or false: 'When an agent is slow and expensive, the most direct, effective fix is to switch to a bigger, stronger model.'

答案：False

A bigger model is often pricier and slower; first optimize architecture + prompt caching (cache reads at 10% price) / batching (50% off) / routing / compression / speculative decoding. Multi-agent already amplifies cost ~15x, so architecture comes first.