Tips for Optimizing LLM Performance: Cut Latency, Cost, and Wasted GPU

Quick Answer: LLM performance optimization comes down to three numbers: time to first token, tokens per second, and how many concurrent requests one GPU can serve. The highest-return fixes in production are prompt caching, continuous batching, KV cache optimization, and quantization. Measure before you change anything, because most slow deployments are bottlenecked somewhere other than where the team assumes.

Almost every conversation about improving LLM performance starts in the wrong place. Someone benchmarks a model, decides it feels slow, and starts pricing bigger GPUs. Two weeks later the bill has doubled and the p99 latency has barely moved.

The reason is that “slow” is not one problem. A 4-second wait before the first token appears is a prefill problem. A response that trickles out at 8 tokens per second is a decode problem. A system that runs fine for one user and falls apart at fifty is a scheduling and memory problem. These have different causes and completely different fixes, and applying the wrong one is how teams end up paying for idle GPU capacity.

We run LLM pipelines inside our own production systems at LYNX, mostly high-volume batch classification and content workflows, plus the AI visibility monitoring we operate for clients. The single biggest cost reduction we have made had nothing to do with hardware. It came from restructuring prompts so the static prefix stayed identical across calls and could be cached. That pattern repeats: the cheapest wins in LLM inference optimization are usually structural, not computational.

This guide covers the techniques that actually move the numbers, in the order we would apply them.

Key Takeaways

• LLM performance splits into two distinct phases, prefill and decode, and each has its own bottleneck and its own set of fixes.
• Time to first token is driven by prompt length, queueing, and prefill compute, while tokens per second during decode is limited primarily by GPU memory bandwidth rather than raw compute.
• Continuous batching at the iteration level is the single largest throughput gain available to most self-hosted deployments, with the original Orca research reporting a 36.9x throughput improvement over request-level batching at equivalent latency.
• Prompt caching cuts both cost and latency on repeated prefixes, and Anthropic reports up to 90% cost reduction and up to 85% latency reduction on long cached prompts.
• Quantization to 4-bit weights reduces memory footprint and speeds up token generation, but it must be validated against your own evaluation set rather than a public benchmark.
• If you want an experienced team handling the search and AI visibility side of your product while your engineers stay focused on inference, LYNX SEO builds organic growth systems for technical companies.

Why Is My LLM Slow? Finding Real LLM Performance Bottlenecks

Every request runs through two phases. 

1. Prefill processes the entire input prompt at once and builds the key-value cache.
2. Decode then generates output tokens one at a time, reusing that cache. 

Prefill is compute-heavy and parallel. Decode is serial and memory-bandwidth-bound, which is why a GPU can sit at low utilization while your users wait.

That distinction explains most confusing benchmark results. A model that looks fast in a single-user test can collapse under concurrency, because the bottleneck moves from compute to memory the moment the KV cache starts competing for space.

NVIDIA’s benchmarking documentation is clear that time to first token includes request queuing, prefill, and network latency, and that longer prompts produce larger TTFT because attention has to process the whole input sequence before generation can begin. If your TTFT is bad, the prompt is the first place to look, not the GPU.

Before touching any configuration, work through the obvious suspects:

• Prompt Length: A 12,000-token system prompt sent on every request is a prefill tax paid on every single call.
• Queue Depth: If requests are waiting for a batch slot, no amount of kernel optimization helps.
• KV Cache Pressure: When cache memory runs out, the scheduler starts preempting or evicting requests, and tail latency spikes.
• Output Length: Decode time scales linearly with tokens generated, so an unbounded max_tokens is a latency bug waiting to happen.
• Cold Starts: Model loading and CUDA graph capture on a scaled-up replica can dominate p99 during traffic spikes.

Four out of five of those are free to fix. That is the point.

How to Measure LLM Performance Before You Optimize It

You cannot optimize what you have not instrumented, and averages will lie to you. Track percentiles, and track them under realistic concurrency rather than a single sequential loop.

The four metrics that matter, plus the cost figure that ties them together:

Throughput and latency trade against each other, so any LLM benchmarking exercise that reports one without the other is describing half a system. Larger batches push throughput up and per-request latency down the queue. The right answer depends entirely on whether you are serving a chat interface or a nightly batch job.

For tooling, NVIDIA’s guide to LLM inference benchmarking concepts covers the differences between benchmarking tools and why two tools can report different TTFT for the same endpoint. Pick one, standardize on it, and re-run it after every change.

LLM Latency Optimization: How to Reduce Time to First Token

TTFT optimization is where most teams find their fastest wins, because prefill work is often entirely redundant.

Prompt caching, sometimes called prefix caching, stores the computed KV cache for a static prompt prefix so subsequent requests skip prefill for that section entirely. If your system prompt, tool definitions, and reference documents never change between calls, you are paying to recompute the same tensors thousands of times a day. Anthropic’s prompt caching documentation describes caching against a prefix hierarchy of tools, then system, then messages, with cached reads billed at a fraction of standard input tokens and a default five-minute TTL that refreshes on each hit. Anthropic reports up to 90% cost reduction and up to 85% latency reduction on long prompts.

The catch is that caching rewards stability. Any change at or before your cache breakpoint invalidates everything after it. Teams that inject a timestamp or a user ID near the top of the system prompt destroy their own hit rate and never realize it.

Three habits that protect the cache:

• Put everything static at the front and everything dynamic at the back, without exception.
• Freeze tool schemas outside of planned release windows, since adding one tool invalidates the prefix for every request that uses tools.
• Log your cache hit rate as a first-class metric, not something you check when the bill arrives.

Beyond caching, prompt length optimization is unglamorous and effective. We have cut system prompts by 40% with no measurable quality drop simply by removing instructions that were written defensively and never tested. Every token you delete is prefill compute you never pay for again.

Improve LLM Throughput With Continuous Batching

Static batching wastes GPU cycles because a batch cannot return until its slowest member finishes. One request generating 2,000 tokens holds the whole batch hostage while nine short requests sit finished and idle.

Continuous batching, also called iteration-level scheduling or in-flight batching, solves this by scheduling at the granularity of a single decode step. Finished sequences leave the batch immediately and new arrivals join on the next iteration. The Orca paper from OSDI 2022 introduced iteration-level scheduling alongside selective batching and reported a 36.9x throughput improvement over NVIDIA FasterTransformer on GPT-3 175B at the same level of latency.

Continuous batching is the closest thing to free throughput in LLM serving optimization, because it changes scheduling rather than model quality, hardware, or output. Every major serving engine now implements some version of it, which means the practical question is not whether to use it but whether your concurrency and batch-size limits are tuned to your actual traffic shape.

Two configuration values are worth real attention. The maximum number of batched tokens per iteration controls how aggressively prefill and decode compete, and setting it too high means a single long prompt stalls everyone else’s token stream. The maximum sequence count controls how much KV cache the scheduler is willing to commit, and setting it too high causes preemption thrashing under load.

KV Cache Optimization: PagedAttention and Memory Management

KV cache is the memory that grows with every token in every active sequence, and it is usually what limits concurrency long before compute does.

The traditional approach reserved a contiguous block of memory sized to each request’s maximum possible length. That produces severe fragmentation, since a request that generates 200 tokens still holds a 2,048-token reservation. The PagedAttention paper borrowed virtual memory paging from operating systems, storing the KV cache in fixed-size blocks that map to non-contiguous physical memory. The result was near-zero waste plus the ability to share cache blocks across requests, and the authors measured 2x to 4x throughput improvement over the serving systems of the day at equivalent latency.

Fixing memory fragmentation raises throughput more reliably than buying a larger GPU, because the constraint was never total capacity, it was how much of that capacity was reachable.

Practical KV cache management beyond paging: cap your context window at what the application genuinely needs rather than the model maximum, use grouped-query attention models where available since they shrink the cache per token, and enable prefix sharing so that requests with a common system prompt point at the same physical blocks instead of duplicating them.

LLM GPU Optimization: FlashAttention, Quantization, and Parallelism

Once scheduling and memory are handled, the remaining gains come from the kernels and the numerics.

1. FlashAttention restructures attention computation to work within the GPU’s fast on-chip memory, tiling and recomputing instead of writing large intermediate matrices to high-bandwidth memory. FlashAttention-2 reports roughly 2x speedup over the original FlashAttention and reaches up to 73% of theoretical maximum throughput in the forward pass, with memory scaling linearly rather than quadratically in sequence length. It is exact attention, not an approximation, so there is no quality tradeoff to reason about.
2. Quantization reduces the precision of model weights, which shrinks memory footprint and, because decode is memory-bandwidth-bound, directly speeds up token generation. AWQ, the MLSys 2024 best paper, showed that protecting roughly 1% of salient weight channels, identified by activation distribution rather than weight magnitude, preserves accuracy at 4-bit precision, with their inference framework delivering more than 3x speedup over an FP16 baseline. INT8 is close to free in quality terms for most workloads. INT4 needs validation.
3. Tensor Parallelism splits individual layers across multiple GPUs so a model too large for one device can still serve. It buys capacity, not efficiency: inter-GPU communication adds overhead, so use the smallest degree of parallelism that fits the model.

Here is how the main techniques compare when you are deciding what to implement first:

Speculative Decoding and Other Ways to Speed Up LLM Inference

Autoregressive decoding is serial by definition: generating K tokens takes K forward passes. Speculative decoding breaks that constraint by having a small draft model propose several tokens, then verifying them in a single pass of the large model.

The original speculative decoding paper from Google Research demonstrated 2x to 3x acceleration on T5-XXL with identical outputs, no retraining, and no architecture changes. The distribution is mathematically preserved, which is what separates this from quality-degrading shortcuts.

Speculative decoding trades arithmetic operations for wall-clock latency, which makes it a good fit exactly when memory bandwidth is the bottleneck and spare compute is sitting idle. In a saturated high-throughput deployment where every FLOP is already committed, it can make things worse. Test it at your real concurrency level, not at batch size one.

Two related moves are worth considering. Chunked prefill interleaves prefill work with ongoing decode steps so a long incoming prompt stops stalling everyone else’s token stream. Structured output constraints, where the grammar restricts valid next tokens, reduce wasted generation on malformed responses that you would otherwise retry.

How to Reduce LLM Inference Costs Without Hurting Quality

Cost optimization and latency optimization overlap, but not perfectly. Some of the biggest savings come from not calling the large model at all.

• Model Routing: Send simple classification and extraction tasks to a small model and reserve the large one for genuinely hard requests. Across our own content pipelines, roughly two-thirds of calls turned out not to need the frontier model once we actually measured output quality per task type instead of assuming.
• Output Length Discipline: You pay per generated token and you wait per generated token. Setting a sensible max_tokens and prompting for concise output is the least sophisticated cost lever available and one of the most reliable.
• Cache Architecture First, Model Downgrade Second: Switching to a cheaper model costs you capability permanently. Restructuring for cache hits costs you an afternoon.
• Right-Sized Hardware: Running a 7B model on an H100 at 20% utilization is a procurement problem, not an engineering one. Match GPU memory bandwidth and capacity to the model and the concurrency target.

Most in-house teams get this far, build a dashboard, and then run out of hours to act on any of it. That is usually the point where bringing in outside help pays for itself, and it is the same calculation companies make when they hand their organic growth program to LYNX instead of hiring four specialists.

LLM Performance Best Practices: The Order to Apply Them

Sequence matters. Optimizing kernels before fixing scheduling is how teams spend three weeks for a 4% gain.

1. Instrument First: Capture TTFT, TPOT, throughput, and cost per million tokens at p50, p95, and p99 under realistic concurrency.
2. Cut The Prompt: Delete unused instructions, move static content to the front, and set a cache breakpoint.
3. Turn On Continuous Batching: Then tune max batched tokens and max concurrent sequences against your traffic shape.
4. Fix Memory: Use a PagedAttention-based engine, enable prefix sharing, and cap context to what the application needs.
5. Quantize And Validate: Start at INT8, test INT4 against your own evaluation set, and never ship on a public benchmark alone.
6. Add Speculative Decoding: Only if decode latency is still the constraint and you have spare compute.
7. Scale Hardware Last: By this point you know exactly what you are buying and why.

Teams that follow this order typically find that steps two through four resolve the complaint entirely, which means the hardware purchase they were about to make was solving a problem they did not have.

LLM Performance vs LLM Visibility: Two Different Problems

Worth separating, because the phrase “LLM performance” gets used for both.

Everything above is about inference performance: how fast your model responds and what it costs to run. A different question, and the one that brings most companies to us, is whether your brand shows up when someone asks ChatGPT or an AI Overview about your category. That is answer engine optimization, and it depends on how your content is structured, cited, and corroborated across the web, not on your GPU configuration.

If that is the version you were searching for, start with our guide to getting more traffic and visibility from LLMs, then work through how to get cited in ChatGPT and how to measure AI search visibility. It is the same discipline we run as a service through our AI visibility and answer engine optimization work.

Optimize the Bottleneck You Have

LLM performance optimization is not a single setting or a single purchase. It is a loop: measure, find the constraint, fix that one thing, measure again. Prompt caching and continuous batching will handle the majority of what most production deployments are complaining about. 

PagedAttention-based memory management, quantization, and speculative decoding cover most of the rest. Bigger GPUs are the last resort, not the first move, and the teams who treat them as the first move are the ones with the worst cost per token.

The discipline is identical to the one that works in search: instrument honestly, fix the constraint that is actually binding, and resist the temptation to buy your way past a structural problem.

If your engineering team is deep in inference work and nobody has time to own organic and AI search visibility, that is what we do. Talk to LYNX SEO and you get the full team, SEO Director, Strategist, Technical Lead, Designer, and Editor, for less than the cost of a single senior in-house hire. Our technical SEO work and cybersecurity and B2B software programs are built for exactly this kind of company.

Related Reading

• How to Get Cited in ChatGPT: Write Pages AI Will Quote
• How to Measure AI Search Visibility
• Shopify AI Overview Optimization: Why You’re Not Showing Up

Sources 

• Kwon, Woosuk et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” 2023. https://arxiv.org/abs/2309.06180
• Yu, Gyeong-In et al. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” USENIX OSDI, 2022. https://www.usenix.org/conference/osdi22/presentation/yu
• Dao, Tri. “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.” 2023. https://arxiv.org/abs/2307.08691
• Leviathan, Yaniv, Matan Kalman, and Yossi Matias. “Fast Inference from Transformers via Speculative Decoding.” 2023. https://arxiv.org/abs/2211.17192
• NVIDIA. “LLM Inference Benchmarking Metrics.” NVIDIA NIM Benchmarking Documentation. 2025. https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html