AI inference cost explorer
Should you call a hosted model API, run an open model on your own GPU, or mix the two? Pick your situation below. The numbers come from a benchmark I ran on AWS in August 2026: same tasks and scoring for every option, with costs calculated from the measured runs at published cloud prices.
Hosted model API
- Got right on this task
- Per 1,000 correct answers
Open 7B model on your own GPU
- Got right on this task
- Per 1,000 correct answers
Monthly cost as volume grows
Source: published runs of the inference cost benchmark, measured on AWS EKS, 17 August 2026. Monthly figures come from its break-even table; quality is what each option scored on this task.
What was measured
Each option answered the same 300 test items three times (900 requests), scored by the same strict rules. Response times are what a user would wait. Costs in this table are for the benchmark runs themselves, with the GPU billed only for the minutes it ran. The monthly figures above assume a GPU kept running all month, which is why the own-GPU numbers differ so much.
| Task | Option | Correct | Per 1,000 correct | Typical wait | Slowest 5% |
|---|
How to read this
- A hosted API charges for every request. Your own GPU costs the same whether it is busy or idle. Volume decides which is cheaper, and how busy you keep each GPU moves the crossover.
- Quality comes before price. On document extraction the 7B model got fewer than half of the records right, so it is not an option there however cheap it looks.
- These are one benchmark's prices: . Your models, prices and traffic will differ, so treat this as a way of thinking about the decision, not a quote.
Want this worked out with your own traffic and prices? I run a fixed five-day assessment that measures your workload the same way and ends with a written recommendation, which may be to keep your current setup.