RAG را با انبار فایل شروع نکنید؛ از ۳۰ سؤال شروع کنید

Most RAG projects start with one big move: dump every file into a warehouse, chunk it, embed it, wire up a model, and wait for good answers. Looks slick from outside. From inside, it hides a hole: you still don’t know which real question the system must answer, how precise it needs to be, or what evidence backs it up.

My fix is simpler: before you build a giant knowledge repo, build a ‘truth kit.’ A small, controlled set of real questions, each one already paired with a desired answer, a citable source, a content owner, a validity date, and a clear acceptance bar.

Does this mean skip architecture, security, system integrations? No. It means don’t build architecture for a problem you haven’t defined yet. Once you know which answers actually matter, every later call on data, retrieval, and model choice gets sharper.

Thirty isn’t a rule; it’s a good starting size. Small enough that your content team can actually review it. Varied enough to expose knowledge gaps fast. Smaller scope? Pick fewer. Multiple units involved? Add questions in waves.

Step one: find real questions, not demo questions that make the model look impressive. Go to the ticket queue, the calls, the sales questions, the internal procedures. Wherever colleagues bounce between files and people just to get an answer, that’s gold. A good question is recurring, time-consuming, sensitive, or expensive.

A hypothetical example from steel: which documents must accompany an order for a given grade? If received specs clash with the original request, who decides? Which version of the maintenance procedure is valid? These are working questions, not chatbot party tricks.

Build an evidence card per question. The question is just the start. Write down: what decision the desired answer enables, how precise it must be, which source is allowed for citation, who owns its correctness, and when to recheck its validity.

Don’t write the desired answer as flat prose; shape its structure too. Maybe it needs a short conclusion, a validity condition, a document reference, a next step. Sometimes the best answer is ‘not enough info, here’s where to go.’ The right silence is still an answer.

Pin the source down: document name, version, section, storage location, approval status. No formal document, and the answer lives only in someone’s head? Don’t hide that. Put the question in ‘needs completion,’ not let the system spit out a confident guess.

The content owner isn’t decoration. This isn’t necessarily the RAG builder; it’s whoever has the authority to confirm or fix that operational truth. Spec or procedure changes? It must be clear who reviews the card and logs the change.

Tag every card one of three ways: answerable, needs completion, or must-not-answer. Answerable: valid source, acceptable answer. Needs completion: the question matters but the document, owner, or criteria are still incomplete. Must-not-answer: an automated reply here could create legal, safety, or unauthorized-decision risk.

This tagging gives you a golden capability: the system can say ‘I don’t know’ usefully. Instead of a vague answer, it states there’s no approved source and points to escalation. In technical and industrial work, that’s a strength, not a weakness.

Now build version zero, but only for this kit. You don’t need every organizational file dumped in. Wire up only the approved sources tied to these questions, configure retrieval, design output so the source and its limits are visible. The goal isn’t completeness; the goal is seeing reality.

Test each question with several phrasings. Real users don’t ask exactly like the evidence card’s wording; they use slang, half-remembered document names, typos. This test tells you whether the failure is retrieval, source quality, question ambiguity, or answer design.

Write acceptance criteria before you see any output, not after. Is the answer relevant? Is its main claim tied to the right source? Does it get the version and date right? Did it drop an important condition? In a prohibited case, did it avoid auto-answering? That’s it.

Using n8n or something similar? Keep the loop light: question in, source retrieved, answer generated, result logged, score given, failure reason noted. A simple data table for questions, source versions, outputs, acceptance states, and reviewer notes is enough. Always keep the ability to compare a new run against the last one.

OpenAI and Anthropic both hammer the same point: evaluate with real-world inputs, user feedback, human review, continuous monitoring. n8n says the same thing: iterative workflow runs, stored evaluation data, catching quality drift. One message: quality isn’t a button, it’s an operational loop.

Tag every failure, don’t just feel vaguely bad about it. Outdated source, source not found, bad chunking, ambiguous question, overly confident answer, incomplete citation, wrong access, needs human escalation. Dump these tags into a data backlog. This is where the small experiment turns into your infrastructure roadmap.

Don’t collect user feedback with just ‘was this good?’ Ask if it moved their decision forward. Did they still have to go dig up the document? Could they verify the source? Did the answer create new risk? That’s the line between readable text and a tool people actually use.

After a few test rounds, you might find the real problem isn’t the model at all. Document versioning is broken, content ownership is unclear, access is scattered, or important questions simply have no official answer. That’s not failure; that’s exactly the point of the truth kit. You saw the real limits of your organization’s knowledge before spending big.

Answerable cards holding steady? Expand scope: add new questions, admit new sources under the same standard, activate ‘needs completion’ items once owners sign off. Every expansion needs re-evaluation, because one change in data or model can shift answers you already trusted.

Reliable RAG doesn’t start by piling up files; it starts by defining testable truth. A small kit of real questions forces you to know what you know, what you don’t, who owns knowing it, and where the system must stay quiet. Move fast, but not on everything at once; start with the questions that will actually move the work.