Most retrieval benchmarks assume a fixed set of documents and a fixed set of right answers. Company knowledge is neither. Prices change, owners change and last quarter’s decision gets reversed.

Questions with an expiry date

We label every evaluation question with the period in which its answer is valid. A retrieval system is scored on returning the answer that was true at the time of the question.

What we measure

Precision at five tells us whether the right memory is near the top. Staleness rate tells us how often a superseded memory outranks its replacement. Citation accuracy tells us whether the source link actually supports the answer.

Reading the results

The surprise was how often a high precision score hid a high staleness rate. A system can find a relevant memory and still find the wrong version of it.