RAG cost guide
Internal Document RAG Cost
A RAG chatbot rarely sends the entire company archive on every question. Cost depends on how much context is retrieved for each answer and how often employees ask questions.
What to estimate
Start with the source library, but do not stop there. A knowledge base with 1,000 PDFs may contain millions of tokens, yet each user question should retrieve only the most relevant chunks. The key estimate is the amount of context sent per question, not only the size of the full archive.
A realistic RAG request includes the user question, system instructions, retrieved passages, citations, conversation history, and the final answer. If each retrieved passage is 500 tokens and the system sends 12 passages, the document context alone is about 6,000 input tokens before instructions and chat history.
Monthly usage example
Suppose 50 employees ask 20 questions each per month. That is 1,000 RAG runs. If each run sends 8,000 input tokens and generates 1,500 output tokens, monthly usage is about 8 million input tokens and 1.5 million output tokens. Multiply those values by the selected model prices to estimate the API budget.
Planning checklist
Estimate the number of active users, average questions per user, retrieved chunks per question, average chunk size, answer length, and expected retries. Then add a safety buffer for longer documents and unclear questions.
How to reduce RAG cost
Remove duplicate pages, split documents into clean sections, improve retrieval quality, limit the number of chunks, cache repeated answers, and use shorter responses for factual questions. For complex workflows, route easy lookup questions to a cheaper model and reserve expensive models for synthesis or final review.
When file size still matters
File size matters during ingestion, summarization, and re-indexing. If you summarize every PDF before storing it, that batch job may be the largest one-time cost. After that, daily RAG usage is usually driven by retrieved context size and answer length.