You Fixed the Prompt. Why Is the Answer Still Wrong?
Most bad AI answers come from missing context, not bad prompts — here's how small teams fix their documents first.
"I've rewritten this prompt a hundred times and the answers are still off." It's the most common complaint we hear. Over the past year or two, the industry's attention has shifted from prompt engineering to context engineering. As models get smarter, the difference in output comes less from how you phrase the instruction and more from what information you hand the model. Instructions change tone; they can't supply facts.
Why doesn't rewriting the prompt fix bad answers?
Most wrong answers happen not because the model is dumb, but because the company information needed to answer never made it into the input. In that case, polishing the instruction is mostly wasted effort. The symptom usually tells you where the real problem sits.
- It states pricing or refund rules confidently and incorrectly → the current policy doc isn't in the context
- The same question gets different answers each time → multiple versions of the source doc exist
- It keeps saying "I can't confirm that" → retrieval isn't finding the right document
- The facts are right but the voice is off → no sample answers to imitate
- It forgets earlier turns in long chats → too much irrelevant material is being stuffed in
Of those five, only the fourth is genuinely a prompt problem.
How do you make company documents AI-readable?
The goal isn't to dump everything into a knowledge base. It's to split content by question and keep exactly one current version. Do this before you shop for tooling.
- Start with the questions. Pull 30 recurring questions from chat logs, call notes, and email.
- One document per question. Don't bundle company overview and terms of service into one file. Keep "Refunds" and "Shipping times" separate, and title them as questions.
- Put the conclusion in the first sentence. A structure that's easy for humans to skim is also easy for a model to quote.
- Tag last-updated dates and scope. If last year's promotion doc is still there, the model will cite it. Delete or archive expired material.
- Collect the failures. Review unanswered questions weekly and add one document each time.
What this looks like in practice
An online course business kept getting inconsistent refund answers from its support bot. The cause wasn't the prompt — it was four refund-related documents scattered across Notion: an old version, an event-specific exception, a draft FAQ, and the current terms. All four were being retrieved at once. The team merged them into one document, split the exceptions into a separate file, and retitled it "Can I get a refund after starting the course?" They never touched the prompt, and wrong answers dropped sharply.
Where should a small team start?
Choosing a vector database or RAG framework can wait. Cleaning up ten documents takes about a day and produces a far bigger improvement.
- Take 20 inquiries from this week and note where the answer actually lives internally
- Write a new document for any question with no source
- For questions with two or more sources, keep one and archive the rest
- Separate sensitive material — salaries, unit costs, customer personal data — out of the knowledge base
- Assign an owner and a monthly review date
Context engineering is less a technology than a housekeeping habit. Models get replaced every few months, but a well-organized knowledge base survives every swap. Better still, those same documents are the raw material that search engines and AI chatbots quote when customers ask about your brand. Before you rewrite the prompt a 101st time, open the document folder.