Why General Knowledge Is Not Enough
Out of the box, a large language model knows a lot. It knows what a refund policy typically looks like. It knows how customer support conversations generally go. It knows what questions people usually ask about software products.
But it doesn't know your refund policy. It doesn't know the specific edge cases your support team handles every week. It doesn't know that your premium plan includes the feature the customer is asking about while your standard plan doesn't — and a confident-sounding wrong answer to that question is worse than no answer.
An AI agent that only knows general knowledge is useful in a limited way. An agent that's been given your specific data — your products, your policies, your processes, your past conversations — is the one that actually replaces the work a trained team member does. This is the plain-English version of how that works.
The Two Ways to Give an Agent Your Knowledge
There's a lot of confusion about what "training" an AI agent even means. In practice there are two distinct approaches, and most production deployments use the second.
Option 1: Fine-Tuning
Fine-tuning means taking a base language model and continuing to train it on your specific data, so the knowledge is baked into the model weights themselves. The model doesn't look things up — it knows them.
Fine-tuning is expensive, demands large amounts of high-quality training data (typically thousands of examples), and has to be repeated every time your data changes. For most business use cases, it's overkill — and we've talked plenty of clients out of it.
Fine-tuning is the right call when you need the model to consistently adopt a very specific tone or style, or when you're in a domain so specialised that general models perform poorly even with good context. Those situations exist; they're rarer than vendors would like you to believe.
Option 2: Retrieval-Augmented Generation (RAG)
RAG is the approach used in the vast majority of production business AI agents. Instead of embedding your knowledge in the model, you keep a separate knowledge base — your documents, policies, FAQs, product information — and the agent retrieves the relevant pieces at query time and uses them to answer.
Think of the difference between a person who has memorised your entire policy manual versus a person who knows where to find the answer and can look it up in seconds. The second approach is more practical, more updatable, and works better at scale.
RAG is what we use for most business AI agent deployments. The rest of this article focuses on RAG because it's almost certainly what applies to your situation.
What Goes Into the Knowledge Base
Your knowledge base is the set of documents and information the agent uses to answer questions. What goes in determines what the agent can answer accurately. There's no clever workaround for this.
Common sources:
Product and service documentation. Features, pricing, plans, specifications, limitations. If a customer asks a detailed question about your product, this is what the agent draws from.
FAQ and support documentation. The answers your team has been giving for years. If you already have a help centre, you're halfway there.
Policies and processes. Return policy, refund eligibility, shipping terms, privacy policy, terms of service. These need to be in plain language that the agent can communicate accurately — more on that below.
Past support conversations. Anonymised resolved tickets are useful — they show the agent what questions actually get asked and how they've been answered well before.
Internal processes. For internal-facing agents: HR policies, IT procedures, onboarding guides, expense processes.
Product catalogues. For e-commerce: product descriptions, specifications, compatibility information, sizing guides.
How to Prepare Your Data
The quality of the knowledge base is the single biggest determinant of how well the agent performs. "Garbage in, garbage out" applies here more strictly than almost anywhere else in technology. We've seen perfectly capable systems produce terrible answers because the source documents were a mess.
Accuracy first. The agent will repeat whatever is in the knowledge base. Outdated information, incorrect details, or contradictory statements will be surfaced to customers as fact. Before loading content, review it. The agent will not catch your old pricing for you.
Plain language. If your policies are written in dense legal language, the agent will reproduce that language. Rewrite key content the way you'd explain it to a customer over the phone.
Completeness for common queries. Map the queries you want the agent to handle and check that each has a clear answer in the knowledge base. The gaps are where the agent fails — and they're rarely where you'd guess.
Chunking. Documents get broken into appropriately sized pieces for retrieval. A 50-page policy document isn't fetched whole; specific paragraphs relevant to the query are. How the document is chunked affects retrieval accuracy more than people expect.
Metadata. Each piece of content benefits from clear metadata: what product it relates to, what type of question it answers, when it was last updated. This helps the retrieval system find the right thing quickly instead of plausibly-relevant-but-wrong things.
What the Retrieval Process Looks Like
When a customer sends a query, here's what happens under the hood:
- The query is converted into a mathematical representation (a vector embedding) that captures its meaning
- The knowledge base is searched for content with similar meaning — not just keyword matching, but semantic similarity
- The most relevant chunks are retrieved — typically the top 3–5
- Those chunks are included in the prompt sent to the language model, along with the customer's query and any conversation history
- The model generates a response grounded in the retrieved content
- The response is returned to the customer
This is why RAG produces more accurate, specific answers than a vanilla language model — the model isn't relying on general training data, it's responding based on your specific content retrieved for that specific query.
The honest caveat: retrieval isn't magic. If the embedding doesn't capture intent well, or the chunks are awkwardly sized, the model can still get the wrong context — and then it produces a confident answer based on the wrong source. This is the failure mode RAG systems live with, and tuning is mostly about reducing how often it happens.
When the Agent Says "I Don't Know"
A well-built agent knows the limits of its knowledge. When a query is outside what's covered — or when the retrieval system doesn't find sufficiently relevant content — the agent should say so and escalate, rather than guess.
This is a feature, not a limitation. An agent that confidently gives wrong answers is more damaging than one that accurately acknowledges uncertainty and routes to a human. We have watched both kinds in production. The confident-and-wrong kind causes harder problems.
The threshold for uncertainty is tunable — how confident does the retrieval need to be before the agent attempts an answer? In high-stakes environments (financial, medical, legal), set it high. For general product queries, lower is fine.
Keeping the Knowledge Base Current
One of the key advantages of RAG over fine-tuning is that updating the knowledge base is straightforward. When your pricing changes, you update the pricing document. The agent reflects the new information immediately — no retraining required.
This is essential for production. Business information changes constantly: product updates, policy revisions, new offerings, seasonal exceptions. A knowledge base that's easy to update is a knowledge base that stays accurate.
Build a maintenance process from the start: who is responsible for keeping it current, how often it's reviewed, and how changes are tested before going live. The projects that drift fastest are the ones where this question was deferred to "later."
What Results to Expect
A well-built, well-maintained RAG knowledge base typically delivers:
- Query answering accuracy: 80–90% of in-scope queries answered correctly in the first month, improving with tuning
- Coverage rate: 60–75% of queries can be answered from the knowledge base without escalation — this improves as gaps are identified and filled
- Escalation quality: When the agent does escalate, it hands the human the customer's question and what was found (or not found) in the knowledge base — context that visibly speeds up resolution
The gap between that initial 80% accuracy and 95%+ is closed through a calibration period — reviewing real conversations, finding where the agent gave wrong or incomplete answers, and patching the knowledge base accordingly. There's no shortcut here. The teams that skip this step plateau at "okay" and stay there.
What You Need to Get Started
Preparing your data for an AI agent knowledge base is often the most time-consuming part of the project — and the most important. Here's what to gather:
- A list of the 30 most common questions your customers or team ask
- Accurate, current answers to each of those questions
- Your key policy documents (returns, refunds, shipping, terms)
- Your product or service documentation
- Any existing FAQ or help centre content
Two or three days of organised document prep before development starts makes the build significantly faster and the initial quality significantly better. We'd much rather slow down at the start than tune our way out of a mess later.
Ready to Build an Agent That Actually Knows Your Business?
An AI agent trained on your data is a different product from a generic chatbot. It answers your specific questions, reflects your specific policies, and represents your business accurately — instead of producing the plausible-sounding industry-average answer.
Talk to us about your business — we'll help you assess your existing content, identify the gaps, and build a knowledge base that makes your agent genuinely useful from day one.