Optimise the system, not only the model call
LLM cost and latency emerge from the entire request path: retrieval, context construction, model selection, tool calls, retries, output length, and infrastructure. Replacing one model can help, but architecture usually offers more durable gains.
Send less, better context
Large prompts often contain duplicated instructions, low-value conversation history, and loosely related documents. Improve retrieval and reranking before expanding the context window. Summarise stable background once, preserve source references, and pass only the information needed for the current task.
Route by task difficulty
Not every request needs the most capable model. Classify simple extraction, formatting, routing, or validation tasks separately from complex synthesis. Use deterministic code where possible and a smaller model where quality remains acceptable.
Routing should be evaluated. A cheaper request that produces unusable output simply moves cost into rework.
Control agent loops
Agent workflows can multiply calls quickly. Set explicit step limits, tool budgets, timeout behaviour, and completion rules. Cache safe tool outputs and avoid asking the model to rediscover information already present in state.
Stream useful output
Streaming does not reduce total generation time, but it can improve perceived responsiveness for user-facing experiences. Pair it with early status feedback when retrieval or tool execution takes meaningful time.
Measure by workflow
Track latency and cost across complete user tasks, not only per API call. Segment by route, model, tool, input size, and outcome. The important measure is the cost of a useful completed workflow.
Protect quality while optimising
Create evaluation cases before changing models, prompts, context length, or retrieval settings. Compare groundedness, completeness, format adherence, and user value alongside speed and cost.
The right optimisation is rarely “make every request cheaper.” It is to allocate capability deliberately, remove unnecessary work, and keep the result inside an acceptable quality boundary.