How to Tell Whether an AI Agent Saw All the Data
TLDR
An AI agent can give a precise answer from an incomplete view. In a public Mage dataset, the first 100 rows contain 27 refurbished purchases, while the full 13,000-row file contains 2,693. Before trusting a total, check whether the agent read the full source, saw only a preview, or stopped early.
Ask an AI agent how many refurbished purchases appear in a file. It replies: 27.
That answer sounds specific. It is also wrong if the agent saw only the first 100 rows.
Mage's public product purchases dataset contains 13,000 purchase rows. In that pinned version of the file, 2,693 rows have refurbished in the condition column. The first 100 rows contain only 27 of them.
The mistake is not bad arithmetic. It is treating what was visible as if it were complete.
Try it with a real dataset
Download the pinned CSV and give it to the tool or agent you want to evaluate.
Ask one question:
How many rows have the condition set to refurbished?
Do not give it the answer first. Ask it to report how many rows it read, whether it reached the end of the file, and whether its answer is exact or only based on a preview.
Here is what the evidence permits:
| What the agent saw | Refurbished rows found | What it can honestly say |
|---|---|---|
| The first 100 rows | 27 | ‘I found 27 in the preview.’ |
| The full 13,000-row file | 2,693 | ‘This pinned file contains 2,693.’ |
| A run that stopped before finishing | Whatever it counted before stopping | ‘I found at least this many, but I do not have the total.’ |
The number on screen can be accurate for the rows shown and still be the wrong answer to the larger question.
Three questions before you trust the answer
1. Did the work finish?
A finished count across the whole file can support an exact answer. A run that timed out, hit a row limit, or was cancelled cannot.
2. How many rows were read?
The file has 13,000 rows. If the agent read only 100, its answer describes those 100 rows. It does not describe the whole file.
3. How many rows were shown to the agent?
Sometimes a tool finishes reading the full source but shows only a short preview. In that case, ask whether the total was calculated from the full file or guessed from the visible rows. A preview is not automatically a random sample.
Ask for a simple receipt
A useful answer should include a short receipt:
-
Source: the exact file or dataset version.
-
Rows read: how much of the source was actually processed.
-
Rows shown: how much reached the agent's visible context.
-
Status: complete, preview only, or stopped early.
-
Claim: exact total, preview count, or lower bound.
For this example, a trustworthy receipt would say: product_purchases.csv, 13,000 rows read, run complete, exact count 2,693.
A reviewer’s 30-second check
Before you reuse an agent’s query analysis, ask:
-
Did execution finish?
-
Did the agent read the full source or only a preview?
-
Is the count exact or a lower bound?
-
What single query would remove the remaining uncertainty?
A complete scan still does not prove that the question was framed correctly or that the source itself contains every record you need. It only makes the boundary clear: the answer came from this complete source, using this condition, at this point in time.
Tools such as Mage Data Explorer and Mage AI Sidekick can help engineers inspect data and run read-only queries. The completeness check described here is a general review pattern, not a claim that those interfaces currently emit every receipt field automatically.
The point is not to distrust every agent answer. It is to ask one basic question before acting on it: did the agent see all the data?
