Tutorial
Create a RAG evaluation dataset from a PDF
Use your own PDF as the source of questions and expected answers. The export is a test file for your stack. It is not a score of a deployed assistant.
Steps
- Create an account and choose a paid plan if you need the evaluation dataset. Free includes JSON of the available types, not the evaluation set.
- Upload the PDF. A scanned PDF needs a paid plan so OCR can read pages with little native text.
- Leave chunking on Balanced unless the strategy guide says otherwise.
- Wait until processing completes, then open the evaluation dataset.
- Export with format
evaluation. Review the questions against the source before you treat them as a gold set.
What to check
Each record should have a question, a ground_truth, and at least one context taken from the document. If a question is not supported by the context, drop it. FAQai attaches source chunk metadata so you can trace the line. Coverage on the document is a review signal, not a production accuracy metric.
Field mapping for a Ragas-style run is on the Ragas integration page.