Lambda and Together AI have been in a price war since Q1. GPU costs for inference dropped 40% year over year. The narrative writes itself: AI is getting cheaper.
It's also getting more wasteful.
The real cost isn't compute
We audited an agent system last month. Their monthly OpenAI bill was $3,200. Their monthly engineering cost to maintain the system was $28,000. The model calls weren't expensive. The architecture that required four redundant calls per user request was.
Every unnecessary chain step, every retry without backoff, every prompt that could be a lookup table: these multiply your costs in ways that GPU price drops don't fix.
What actually reduces cost
Three patterns that consistently cut agent operating costs by 60-80%.
Caching at the semantic level. Not HTTP caching. Embedding-based caching. If a user asks the same question with different words, you don't need another model call. A cosine similarity check against your last 1,000 responses costs fractions of a cent.
Right-sizing models per step. Not every agent step needs GPT-4. Classification? Use a fine-tuned small model. Extraction? Regex might work. Generation? That's where you spend on the big model. Most agent pipelines use their most expensive model for every step because nobody bothered to profile which steps actually need it.
Failing fast on bad inputs. A $0.03 model call that's going to fail anyway is still $0.03 wasted. Input validation, intent classification, and scope detection before the expensive call are not just reliability features. They're cost controls.
Cheaper GPUs help. But the teams spending the least on AI infrastructure aren't the ones with the cheapest compute. They're the ones who designed their architecture to make fewer, smarter calls.
If your agent costs keep climbing even as GPU prices drop, we should look at the architecture.