Cross-model validation, investigating disagreements, and converging above 90% agreement.
7 · Validation & the AI Judge
Extraction alone is not trust. Sci-database's answer to "how do I know my database is accurate?" is cross-model validation: an independent AI model re-extracts a sample of your papers from scratch, and the platform shows you exactly where the two models agree and disagree — field by field, record by record. This mirrors the dual-extraction practice of Cochrane-style systematic reviews, with models instead of grad students.
Starting a validation
In the Database tab, click Validate on any completed job. The Validate tab opens with the setup controls:
Validation Sample — a slider from 1–100% (default 20%) of the papers that produced data. You don't need to validate everything to know your accuracy: a random sample is how audit works.
Validation Model — choose the second reader:
- Google Gemini (Gemini 3.7 Flash, 3.6 Flash, 3.5 Flash-Lite, 3.1 Flash Lite, 2.5 Flash, 2.0 Flash Lite) — available with system credits (2 credits/paper) or 100% free with your own Gemini API key.
- GPT-4o-mini / GPT-4o (OpenAI) — bring your own API key (100% free).
- Claude 3.5 Sonnet (Anthropic) — bring your own API key (100% free).
When using third-party models or custom keys, an API key dialog safely prompts for your key: "This key will only be used for this validation session and will not be stored."
Start Validation — uses 2 credits per sampled paper with our system AI, or is FREE if you provide your own API key. Progress shows as "Validating i/N".
The validator model receives the same full paper text and your same schema, but with its own even stricter instructions (analyze the full document, never rely on the abstract alone; no abbreviations; maximum specificity; exact numerical precision; N/A when absent).
Reading the results
The header shows the overall agreement percentage across all validated fields, and each paper gets its own agreement badge. Green check = full agreement; red triangle = disagreements found. A field counts as a disagreement only when the validator returned a different non-empty value — if the validator found nothing for a field, that's marked neutral (ℹ), not a conflict.
Expand any paper to see the Record Comparison table:
| Column | Content |
|---|---|
| Field | Your schema field |
| Original (Kateeb) | What the first extraction found |
| Status | ✅ match · ⚠️ mismatch (row highlighted red) · ℹ️ validator found nothing |
| Validated (model) | What the second model found |
Nothing is hidden and nothing is auto-corrected: every red row is a question the platform is putting to you.
The AI Judge: investigating disagreements
For any paper below 100% agreement, click Investigate (1 credit per investigation). A third AI acts as an expert adjudicator: it re-reads the full paper text for each disputed field and must return either one of the two candidate values, a value it can point to directly in the text, or N/A — plus a one-to-two sentence reasoning.
The result appears inline as an Investigation Result row with the final value and the arbiter's reasoning. From then on, the arbiter's resolved value becomes the reference for that field: the agreement score is recalculated against it.
You can download the full comparison as Report — a CSV (validation_report_job_….csv) with columns Paper Title, Record Index, Field Name, Original Value, Validated Value, Is Match — ready to attach to a manuscript's supplementary materials.
Closing the loop: Improve Schema
Most disagreements are not model failures — they're ambiguous field descriptions ("efficiency" — of what? initial or stabilized? which unit?). Click Improve Schema (1 credit) and the AI analyzes the investigated discrepancies and proposes better field descriptions (names and types are never touched).
Each suggestion shows the reasoning, the old description (red) and an editable suggested description (green); you Accept & Edit or Reject per field. Accepted changes land in your schema editor — save the schema, then re-run the extraction.
Converging to 90%+
This is the trust loop the platform is designed around:
extract → validate (2nd model) → investigate (AI Judge) → improve schema → re-run
▲ │
└────────────────────────────────────────────────────────────────────────────────┘
A typical trajectory over the passes: ~70% agreement on the first raw run (models disagree on units and ambiguous fields) → ~85% after refining the schema → 90%+, the platform's trust threshold, at which point disagreement is confined to genuinely hard judgment calls that you have personally adjudicated. Your database is now something you can defend in peer review — with the validation report to prove it.