Verifying product data without ground truth

01 · The problem

An extraction that doesn't say when it's wrong

Nricher enriches online retailers' catalogs. From a product's title and description, a pipeline categorizes it, extracts its attributes (color, dimensions, material, weight…) with regular expressions, normalizes the values and produces a structured product sheet. This extraction is deterministic and easy to inspect, but it gives no confidence score.

So checking is manual: an account manager reviews a sample of 1,000 products, which takes one to two days. And the difficulty is real. On a sofa bed, the extraction reads "140 × 190": those are the mattress dimensions. It reads a weight of 32 kg: that is the foam density. The client's catalog has its own errors too.

The task is not to compare a prediction with the truth, but to arbitrate between two unreliable sources.

Threshold

Fig. 1 — The threshold is the only knob at deployment time: to the left only the safe values are validated, to the right more values ship and more errors slip through. The marked point at 68% is the one kept in the thesis: beyond it, precision falls below 99%.

02 · The reference

Building a ground truth without annotating anything

When the catalog and the extraction agree after normalization, I treat the value as correct.
That set then contains only agreements. I inject errors into 20% of rows by replacing the value with another real value of the same attribute: an obviously wrong value would teach nothing. The seed is fixed, so everything is reproducible.
I add an "information missing" case by removing the information from the text for about 10% of rows of each attribute.

The benchmark covers 4 production catalogs, 35 attributes and 30,919 rows. A reference point: 73.3% of values are already correct without doing anything. Any method has to beat that.

03 · The sensor

Textual inference, sentence by sentence

The value to check becomes a hypothesis, for example "the height is 84". The multilingual mDeBERTa-v3 model (the catalogs are in French) tests it against each sentence of the description and returns three probabilities: entailment, contradiction or neutral. Descriptions are split into sentences (21 at the median), because these models degrade on long texts, and I keep the maximum: a single supporting sentence is enough.

At the best threshold (0.70 on entailment), this rule reaches 0.808 balanced accuracy and 0.919 precision.

Looking at attributes one by one, I found what makes them hard to verify. It is not the number of possible values, it is negation: attributes expressed with a negative sentence ("cover not removable") plateau between 0.437 and 0.854 AUC, while the others reach 0.968 to 0.997.

04 · The decision

A tabular model you don't train

TabPFN is pretrained on millions of synthetic datasets, then frozen. You don't train it: it receives labeled examples in its context and predicts in a single forward pass. It is light enough for thousands of runs on a laptop without a GPU, about 4 seconds each. Since it cannot read text, it receives the description encoded with BGE-M3 and reduced to 50 components by PCA, with or without the inference scores; one model per attribute.

trained model with inference scores trained model without the scores rule on the scores, no training

TabPFNembeddings + inference scores · 55 features 0.925

TabPFNinference scores only · 5 features 0.906

TabSTARraw text + inference scores 0.898

TabPFNembeddings only 0.836

Inference rule0.70 threshold, no training 0.808

TabSTARraw text only 0.742

Fig. 2 — Mean balanced accuracy over the 35 attributes. The three trained models that receive the inference scores come first; TabPFN with only those 5 features stays within 2 points of the best model, which uses 55.

The paired ablation (Wilcoxon test) is clear: removing the 4 inference scores costs 8.9 points, removing the 50 embedding components costs only 1.9. Four numbers computed against the value to check are worth more than fifty components computed blindly: what matters is how the text representation is oriented, not how rich it is. The standard deviation across attributes also drops from 0.128 to 0.079: the model is both better and steadier.

TabSTAR let me test the opposite: a model pretrained on 350 real datasets, with a trainable text encoder, that reads the raw description. On its own it does worse than the inference rule (0.742). With the scores it gains 15.6 points (0.898) but stays behind TabPFN, while taking about 28 times longer (116 s per run, on GPU).

05 · In production

One setting: the threshold

The architecture I proposed plugs into the existing pipeline without changing it:

Product title and descriptionexisting pipeline: categorization, rule-based extraction, normalization
Do the client's catalog and the extraction agree?

Yes → the row joins the training corpus. The system feeds on its own flow, with no annotation.

No → the row goes to verification.
Verificationinference scores (mDeBERTa-v3) + encoded description (BGE-M3, PCA) + code of the candidate value → TabPFN → probability that the value is correct
Is the probability above the threshold?

Yes → the value goes straight into the product sheet.

No → review by a data developer.

The threshold moves without retraining anything. Set for 99% precision, it automatically validates 68% of values: human review now covers only a third of the volume, and that third is selected, not drawn at random.

06 · Limitations

What this system does not see

The reference comes from agreement between two sources. If the catalog and the extraction share a bias, it goes unnoticed. A sample should be annotated independently of the text.
The errors are injected, not observed. Misattributing a value, the most frequent error in production, is not reproduced. The next step is to validate on real errors.
Detecting an absence is not verifying. Without a candidate value there is no hypothesis, hence no score: in the "information missing" case, accuracy plateaus at 0.57. That case deserves a task of its own.