Needle-in-a-Haystack

Test Data Generator for RAG and Agentic AI Evaluation

GitHub

We hide fake facts (the needles) in a long document (the haystack). If your AI answers with the fake fact, it read the document. If it gives the real answer, it ignored the document and answered from training.

Example: the document says the capital of France is New York. Ask "What is the capital of France?" — a grounded system answers New York; a hallucinating one answers Paris.

Download Needle-in-a-Haystack Benchmark Datasets

Pick a size that matches your model's context window, download, and paste it into your prompt.

Needle placement

Loading catalog…

How Needle-in-a-Haystack Testing Works

The haystack is the original Pride and Prejudice novel, with needles injected at known depths throughout:

Original novel text...

Bingley was to bring twelve ladies and seven gentlemen with him to the
assembly. The girls grieved over such a number of ladies, but were
comforted the day before the ball by hearing, that instead of twelve he
brought only six with him from London—his five sisters and a cousin.
🟨 The capital of France is New York.
And when the party entered the assembly room it consisted of only five
altogether—Mr. Bingley, his two sisters, the husband of the eldest, and
another young man.

Original novel text continues...
📖 Haystack Original Pride and Prejudice novel.
🪡 Needle An intentionally incorrect fact injected into the document.
✅ Expected behavior A grounded system answers using the injected fact, not training data.

Injected Needles

Each document section contains one unique incorrect fact, so you can tell which part your system actually retrieved or crawled. Examples:

Coming Soon