Launch capacity guide
Token Rate Limit and API Capacity Guide
A workload can be affordable but still fail at launch if it exceeds requests per minute, tokens per minute, or concurrent processing capacity. Estimate rate limits alongside monthly cost.
Track two limits, not one
AI API providers commonly limit both request count and token volume. A short classification job may hit requests per minute first, while a document summarizer, RAG answer, or code review may hit tokens per minute first. Capacity planning should estimate both dimensions before traffic arrives.
Estimate tokens per minute
Start with the token count for one successful action: input tokens plus expected output tokens. Multiply by peak requests per minute, workflow rounds, and retry overhead. Use the peak minute, not the monthly average, because rate limits are enforced over short windows.
Peak-minute example
A PDF summary workflow sends 18,000 input tokens and returns 2,000 output tokens. If five users submit a document in the same minute and each job needs two model calls, the burst can consume about 200,000 processed tokens before retries.
Separate user traffic from background jobs
Batch imports, nightly summaries, embedding jobs, and re-indexing can compete with user-facing requests. Schedule background work outside peak usage, cap concurrency, and reserve capacity for interactive actions that users are waiting on.
Reduce bursts before upgrading limits
Trim oversized prompts, split large documents into queued chunks, cache stable prefixes, deduplicate repeated context, and use smaller models for routing or extraction. These changes reduce both cost and rate-limit pressure. For repeated prefixes, compare the result with the prompt caching cost guide.
Validate with provider dashboards
After a representative load test, compare estimated requests per minute, tokens per minute, failures, retries, and latency with the provider dashboard or API usage fields. Confirm current limits and increase-request rules in the official OpenAI rate limits guide, Anthropic rate limits guide, and Gemini rate limits guide.