RAG with your own documents: check retrieval and citations before trusting an answer
A practical retrieval-augmented generation checklist with an invented policy example, expected source passages, and separate checks for retrieval and answers.
In this article
Choose the files deliberately
Retrieval-augmented generation (RAG) adds relevant retrieved material to a language model's input. The LlamaIndex introduction distinguishes loading, indexing, storing, querying, and evaluation.
Start with a small collection. The directory reader supports file selection and custom metadata. Record the owner, version, and effective date as editorial checks; modification time alone need not settle which policy applies.
Define a checkable test
Invented exercise: Travel-2026.md sets a 14-day train-receipt deadline; Travel-archived.md says 30 days. Ask about an October 2026 trip. Expect 14 days and the current passage. These fictional rules are not test results.
Also ask whether guests can claim tickets. Neither fictional file covers guests: require the answer to identify that information gap instead of inventing eligibility.
Separate retrieval from the answer
Save retrieved passages. Did the current passage appear? Retrieval evaluation compares retrieved nodes with expected ones. If absent, investigate selection or search.
Then check the response against its evidence. Response evaluation includes a faithfulness check against retrieved context. Keep the question, passages, and answer together so someone else can inspect the mismatch.
Open the citation
Check every deadline, condition, and exception against the cited passage. The CitationQueryEngine example demonstrates opening the actual source text.
Explore LlamaIndex, IBM’s RAG video, and agentic RAG.
Original CurioFuse editorial. A guide to explore and evaluate AI, without an implied product test.
