How It Works

Understanding the validation process and design decisions.

Overview

linkml-reference-validator validates that quoted text (supporting text) actually appears in cited references. It uses deterministic substring matching rather than fuzzy or AI-based approaches.

The Validation Process

1. Text Normalization

Before matching, both the supporting text and reference content are normalized:

Lowercased: "MUC1" → "muc1"
Punctuation removed: "c-Abl" → "c abl" (a point or comma between digits is kept, so "10.5" stays one number)
Meaning-bearing symbols kept as words: < > ≤ ≥ become lt gt le ge, ± (or +/-) becomes pm, and a minus sign before a number stays on it, so "p < 0.05" does not match "p > 0.05" and "2.5 fold" does not match "-2.5 fold". ASCII and Unicode spellings are equivalent (<= and ≤, - and −). Publisher spellings count too: MEDLINE's >/= and </= for ≥ and ≤, a middle-dot decimal (57·4 is 57.4), and an en dash or Unicode hyphen printed as a minus sign (z score, –1.98) or inside +/–. A − set off from its number by a space (from − 2.92) is still a sign, unless a number comes before it (10 − 2 is a subtraction). A hyphen or en dash after a word or digit (IL-6, 3-5, 1–12) is still punctuation. A dash after a space and before a digit is a sign, so a range spaced on one side only (18 –65) reads as 18 and -65; quote it as the source prints it. A middle dot after a letter-joined number is not a decimal point, so MgCl2·6H2O keeps its hydrate dot (and pH7·4 does not read as pH7.4; pH 7·4 does).
Whitespace collapsed: Multiple spaces become single space
Editorial notes removed: "[mucin 1]" → ""

Example:

Original: "MUC1 [mucin 1] oncoprotein blocks c-Abl!!!"
Normalized: "muc1 oncoprotein blocks c abl"

This allows matching despite formatting differences while maintaining exactness.

2. Substring Matching

After normalization, the validator checks if the supporting text appears as a substring in the reference content.

Simple case:

supporting_text = "MUC1 oncoprotein"
reference_content = "...The MUC1 oncoprotein blocks nuclear..."
# Match: "muc1 oncoprotein" found in normalized reference

Matches start and end on whole words:

A quote may not begin or end inside a word of the reference. Otherwise a quote stating the opposite of the paper would pass:

supporting_text = "activates AKT signaling"
reference_content = "...Loss of PTEN inactivates AKT signaling..."
# NOT FOUND: "activates" only occurs inside "inactivates"

supporting_text = "5 mM H2O2"
reference_content = "...treated with 10.5 mM H2O2..."
# NOT FOUND: "5" only occurs inside "10.5"

Each part of a quote split with ... is held to the same rule.

One allowance: when a quote ends in a letter, the reference word it ends on may carry trailing digits. Full-text extraction welds superscript citations onto the word before them, so tumor suppressor<sup>12</sup> reads as suppressor12, and a quote of PTEN is a tumor suppressor must still match. A quote ending in a digit gets no allowance (10 never matches 105). The same allowance means a quote ending in AKT matches AKT1, as plain substring matching always did.

Because a point or comma between digits stays part of the number, numbers must be quoted with the reference's own separators: days 3, 5 and 7 does not match days 3,5 and 7, and 10 000 does not match 10,000.

Empty excerpts are rejected:

An empty or whitespace-only excerpt is always an error, never a pass. The empty string is a substring of every document, so a blank excerpt would otherwise be reported as verified evidence while quoting nothing at all:

supporting_text = ""     # or "   ", "\n"
# ERROR: Supporting text is empty

The check runs before the reference is fetched, and applies even when the reference prefix is on the skip list: a blank excerpt is a defect in the data regardless of what it cites. An excerpt slot that is absent is a different matter — whether it is required is up to your schema.

Minimum excerpt length (optional):

A blank excerpt is the extreme case. A two-character one is barely better: a short string matches almost any paper by chance, so the match succeeds without the excerpt being real evidence. Set min_excerpt_length to require a minimum:

# .linkml-reference-validator.yaml
validation:
  min_excerpt_length: 20

Keep it under validation:. A config file that has a validation: section ignores validation keys placed at the top level — silently, which for an opt-in check means it looks enabled while never running.

The check is off by default (0), so existing data keeps validating as before. Length is measured in non-whitespace characters of quoted text:

Whitespace is ignored, so PDF-to-text extraction damage does not change the count — "t he pro tein" and "the protein" both measure 10.
No tokenizing, so chemical nomenclature and other punctuation-dense strings are counted in full — "2,3-dihydroxybenzoate" measures 21, not 2 words.
Editorial [brackets] and ... separators do not count, since they are never matched against the reference.

Like the blank check, this runs before the reference is fetched.

3. Ellipsis Handling

When supporting text contains ..., each part is matched separately:

Supporting: "MUC1 oncoprotein ... nuclear targeting"
Parts: ["MUC1 oncoprotein", "nuclear targeting"]
# Both parts must exist in the reference

4. Title Validation

In addition to excerpt/quote validation, the validator can verify reference titles using exact matching (not substring). Titles are validated when:

A slot implements dcterms:title or has slot_uri: dcterms:title
A slot is named title (fallback)

Example:

reference_title: "MUC1 oncoprotein blocks nuclear targeting of c-Abl"

Title matching uses the same normalization as excerpts (case, whitespace, punctuation, Greek letters) but requires the entire title to match, not just a substring.

# These match after normalization:
expected = "Role of JAK1 in Cell-Signaling"
actual = "Role of JAK1 in Cell Signaling"
# Both normalize to: "role of jak1 in cell signaling"

# These do NOT match (partial title):
expected = "Role of JAK1"  # Missing "in Cell Signaling"
actual = "Role of JAK1 in Cell Signaling"

See Validating Reference Titles for detailed usage.

Why Deterministic Matching?

Not Fuzzy Matching

We explicitly avoid fuzzy/similarity matching because:

Accuracy: No false positives from "close enough" matches
Reproducibility: Same input always gives same result
Explainability: Clear why something matched or didn't
Trust: Critical for scientific accuracy

Not AI-Based

We don't use LLMs or semantic similarity because:

Determinism: Results must be reproducible
Verifiability: Humans can verify the match themselves
No hallucinations: The text either exists or doesn't
Simplicity: No model dependencies or API costs

Reference Fetching

The validator uses a plugin architecture to support multiple reference sources. Each source type is handled by a dedicated plugin that knows how to fetch and parse content from that source.

PubMed (PMID)

For PMID:12345678:

Queries NCBI E-utilities API
Fetches abstract and metadata
Parses XML response
Caches as markdown with YAML frontmatter

PubMed Central (PMC)

For PMC:12345:

Queries PMC API for full-text XML
Extracts all sections (abstract, introduction, methods, results, discussion)
Provides more content than abstracts alone
Also cached as markdown

DOI (Digital Object Identifier)

For DOI:10.1234/example:

Queries Crossref API
Fetches metadata and abstract (when available)
Caches as markdown

Local Files

For file:./path/to/document.md:

Reads file from local filesystem
Extracts title from first markdown heading (or uses filename)
Content used as-is (no parsing for HTML files)
Caches to allow consistent validation

Path resolution: - Absolute paths work directly - Relative paths use reference_base_dir config if set, otherwise current directory

URLs

For url:https://example.com/page:

Fetches page via HTTP GET
Extracts title from <title> tag (for HTML)
Content preserved as-is
Cached like other sources

Caching

References are cached in references_cache/ as markdown files:

references_cache/
  PMID_16888623.md
  PMC_3458566.md

Cache file format:

---
reference_id: PMID:16888623
title: MUC1 oncoprotein blocks nuclear targeting...
authors:
  - Raina D
  - Ahmad R
journal: Molecular Cell
year: '2006'
doi: 10.1016/j.molcel.2006.04.017
content_type: abstract_only
---

# MUC1 oncoprotein blocks nuclear targeting...

**Authors:** Raina D, Ahmad R, ...
**Journal:** Molecular Cell (2006)

## Content

The MUC1 oncoprotein blocks nuclear targeting...

An entry with no title has no # title, authors or journal lines, but it still gets the ## Content marker. The content is everything after the first exact ## Content line of that header, so a heading in the text such as ## Contents of the kit stays part of the content. A file written before untitled entries got the marker is read whole.

A cache file whose frontmatter is empty, isn't valid YAML, or isn't a mapping is treated as missing. Validation re-fetches the reference and rewrites the file, and tools that walk the cache skip the file with a warning.

Cache Benefits

Offline usage: Work without network after initial fetch
Performance: Instant validation after first fetch
Reproducibility: Same reference version for all validations
Inspection: Human-readable cache files

LinkML Integration

The validator is a LinkML plugin that uses special slot URIs:

classes:
  Statement:
    attributes:
      supporting_text:
        slot_uri: linkml:excerpt  # Marks as quoted text
      reference:
        slot_uri: linkml:authoritative_reference  # Marks as reference ID
      reference_title:
        slot_uri: dcterms:title  # Marks as reference title (optional)

When LinkML validates data, it calls our plugin for fields marked with these URIs.

The plugin discovers fields via: - implements attribute (e.g., implements: [dcterms:title]) - slot_uri attribute (e.g., slot_uri: dcterms:title) - Fallback slot names (reference, supporting_text, title)

Besides the URIs above, a URI of your own counts as a reference or excerpt field when its local name ends in the word. One Field, Slot or Value suffix may follow it, and for references also an identifier suffix such as Id or Uri. So ex:reference, ex:cross_reference, ex:myReferenceField and ex:reference_id are reference fields, and ex:myExcerptField is an excerpt field. ex:referenceType, ex:excerpt_location and ex:excerpt_id describe a reference or excerpt, so they are not. If a field of yours is missed (say ex:excerpt_text), declare it with implements or one of the URIs above.

Editorial Conventions

Square Brackets `[...]`

Used for editorial clarifications inserted into quotes:

Original reference: "MUC1 oncoprotein blocks nuclear targeting"
Your quote: "MUC1 [mucin 1] oncoprotein blocks nuclear targeting"

The [mucin 1] is removed before matching.

Ellipsis `...`

Used to indicate omitted text between parts:

Original: "MUC1 oncoprotein blocks nuclear targeting of c-Abl"
Your quote: "MUC1 oncoprotein ... c-Abl"

Both parts must exist in the reference.

Design Principles

1. Conservative by Default

Only exact substring matches count
No approximations or suggestions
Fail fast on mismatches

2. Progressive Disclosure

Simple cases require minimal syntax
Advanced features (editorial notes, ellipsis) available when needed
Sensible defaults (cache location, etc.)

3. CLI-First

Command-line is the primary interface
Python API available for integration
No GUI required

4. Standards-Based

Uses LinkML schemas
NCBI standard identifiers (PMID, PMC)
Markdown for cache files

Limitations

What This Tool Does NOT Do

Semantic matching: Won't match paraphrases
Citation formatting: Not a bibliography manager
Fact checking: Only verifies text existence
Plagiarism detection: Not designed for that purpose

Known Limitations

Abstracts only for most PMIDs: Full text requires PMC. When validation fails and only an abstract was available, the error message will note this - the excerpt may exist in the full text.
Network required: For initial reference fetch
English-focused: Normalization optimized for English text
No OCR: Can't extract text from images/PDFs in papers

How It Works

Overview

The Validation Process

1. Text Normalization

2. Substring Matching

3. Ellipsis Handling

4. Title Validation

Why Deterministic Matching?

Not Fuzzy Matching

Not AI-Based

Reference Fetching

PubMed (PMID)

PubMed Central (PMC)

DOI (Digital Object Identifier)

Local Files

URLs

Caching

Cache Benefits

LinkML Integration

Editorial Conventions

Square Brackets `[...]`

Ellipsis `...`

Design Principles

1. Conservative by Default

2. Progressive Disclosure

3. CLI-First

4. Standards-Based

Limitations

What This Tool Does NOT Do

Known Limitations

When to Use This Tool

Good Use Cases ✅

Not Recommended ❌

How It Works

Overview

The Validation Process

1. Text Normalization

2. Substring Matching

3. Ellipsis Handling

4. Title Validation

Why Deterministic Matching?

Not Fuzzy Matching

Not AI-Based

Reference Fetching

PubMed (PMID)

PubMed Central (PMC)

DOI (Digital Object Identifier)

Local Files

URLs

Caching

Cache Benefits

LinkML Integration

Editorial Conventions

Square Brackets [...]

Ellipsis ...

Design Principles

1. Conservative by Default

2. Progressive Disclosure

3. CLI-First

4. Standards-Based

Limitations

What This Tool Does NOT Do

Known Limitations

When to Use This Tool

Good Use Cases ✅

Not Recommended ❌

Square Brackets `[...]`

Ellipsis `...`