verifold

VALIDATION EVIDENCE

Observed.
Not assumed.

What we tested, what we found, and what remains unknown.

These are curated summaries of recorded local experiments, not live reports, customer incidents or production guarantees.

Request an engineering pilot

Contact: hello@verifold.xyz

M2 / CONTROLLED PIPELINE

Recorded 27 October 2026

Present in the index.
Missing from the query.

An intact index does not establish that a particular serving route can return the expected document.

100declared local HTML pages
300expected indexed points
0 / 1query expectations met during fault

The setup

A controlled local pipeline fetched HTML, retained capture and extraction outputs, produced chunks and deterministic lexical embedding artifacts, wrote real Qdrant points, and served record lookups and search requests. A separate Rust verifier read the supported artifacts and issued real probes.

The observed failure

A deliberately altered serving route excluded the expected document. All 100 live documents matched the indexed metadata and all 300 direct lookups matched, but the declared top-1 query did not return its expected result. The last matching stage was INDEX; the next observed gap was RETRIEVAL.

OBSERVED BOUNDARY

Index matches → retrieval expectation missed

Global verdict: UNKNOWN · Scope: one local recipe and declared query

The recovery

Restoring the serving route made the same query expectation succeed. Separate controlled cases also exercised extraction gaps, stale versions, missing index points and failed deletion propagation.

What this does and does not establish

The experiment localizes a discrepancy along one bounded reference pipeline. It uses synthetic local content and deterministic lexical vectors—not a learned embedding model. A missed query does not establish that the document is universally unsearchable. A passing probe does not establish semantic relevance, complete recall, AI answer correctness or future synchronization.

A purpose-built integration test issuing this same query can detect the same failure. We have not established superior detection or lower integration effort.

M3 / EXTERNAL STACK

Recorded 27 October 2026

A healthy retry.
An unhealthy deletion.

Independent source/index evidence caught a retained point after an application retry returned normally.

LlamaIndex core 0.14.25Qdrant adapter 0.10.4FastEmbed 0.9.1

The setup

An unmodified LlamaIndex ingestion stack processed four local URLs with real Qdrant and a local learned embedding model. The application supplied an explicit source inventory, document versions, retained tombstones and a one-chunk expectation.

  1. Deletion rejected. A deliberately read-only Qdrant key caused the deletion request to fail with a native 403 error.
  2. Failed state persisted. The application explicitly saved the mutated document-store state after catching the exception, then reloaded it.
  3. Retry returned normally. Restoring write access and retrying produced zero new nodes. The source-deleted page remained indexed and searchable.
  4. Difference independently observed. The existing Rust observer joined the version-2 source tombstone with the retained version-1 indexed point and recorded recurrence.
OBSERVED MISMATCH

Source: deleted, v2 · Index: retained, v1

Global verdict: UNKNOWN · Recurrence is investigation evidence

The recovery

The pipeline owner explicitly removed the retained point. A deletion probe and a positive probe for an updated document then behaved as expected. Verifold observed the recovery; it did not repair the data.

Conditions and limitations of this result

Persisting the mutated document store after the caught error was an explicit application action. Automatic crash persistence, an older restored checkpoint and arbitrary LlamaIndex deployments were not tested. The source ledger and expected inventory were application-provided contracts.

Extraction and chunk manifests were retained as upstream claims. No general independent verification of learned embedding execution or indexed vector bytes was demonstrated. The native 403 already exposes the initiating failure; a targeted Qdrant absence assertion can also detect the leftover point.

This is one controlled external-stack integration, not an independently reported customer incident. Customer adoption, investigation speedup and commercial value remain unproven.

LOCAL ASSURANCE / 20 QUESTIONS

Recorded 27 October 2026

Measure retrieval.
Assess source data separately.

The existing M4 developer beta now has a local retrieval-evaluation and source-assessment workflow in the same explorer.

The actual execution

Twenty frozen excerpts from Verifold's real documentation were ingested through the supported LlamaIndex pipeline into real Qdrant, with PostgreSQL metadata and a pinned local MiniLM embedding model. Twenty distinct operational questions and answer-bearing relevance labels were declared before search execution.

1.00 → 0.90Recall@3 after two indexed documents were removed
1.00Recall@3 after owner repair
20queries retained across every condition

Healthy / missing / repaired MRR@3 was 0.925 / 0.825 / 0.925. Precision@3 was 0.3333 / 0.3000 / 0.3333; nDCG@3 was 0.9446 / 0.8446 / 0.9446. Each question has one directly answering excerpt, so healthy Precision@3 is one third. These are macro averages over all 20 questions, including controls.

Search success did not erase source-quality findings

Separate faults made four of 20 source capture artifacts unavailable, left empty or malformed processed text, introduced an exact source-content duplicate, and supplied an outdated timestamp, a missing timestamp and a missing mandatory content property. The assessment identified those conditions while Recall@3 remained 1.00. Missing evidence was shown explicitly, not scored as healthy.

SEPARATE EVIDENCE CONTRACTS

Integrity remains UNKNOWN, even when retrieval succeeds.

No universal health PASS or opaque data-quality score.
Evidence provenance and limitations

The implementation agent assigned source-derived relevance labels before retrieval; no independent human or customer judgment is claimed. The corpus is real repository documentation with controlled faults, not customer traffic. Equivalent native queries and targeted integration assertions detected the same faults. This demonstrates a reproducible local workflow, not superior detection, general search improvement or commercial demand.

M4 provides bounded preflight and read-only observation for one supported LlamaIndex/PostgreSQL/Qdrant topology. Source authority, retained deletion history, versions and expected chunks must come from upstream. Source-artifact coverage does not prove a crawler visited every page. Transformation lineage and source timestamps remain declared claims where not independently inspected. Training-data intelligence is research only.

FROZEN LOCAL RESULTS / 27 OCTOBER 2026

What the measurements say.

Six aggregate views regenerated from preserved evidence. Controlled local workloads, no customer validation. Parity and regressions are part of the result.

Retrieval changes after omission and owner repair

Retrieval changes after omission and owner repair. Exact values in the following table.
Recall@3 — exact aggregate values
Condition Recall@3 MRR@3 nDCG@3
Healthy 1.000000 0.925000 0.944639
Missing 0.900000 0.825000 0.844639
Repaired 1.000000 0.925000 0.944639
20 documentation excerpts, 20 declared questions; one positive judgment per query, assigned by the implementation agent before execution. Real PostgreSQL, Qdrant and local MiniLM embeddings; observer bound: one CPU / 768 MiB. These are metric observations, not CPU or timing measurements. All 20 queries retained; two documents removed, then restored by the fixture owner. One execution per condition, no population confidence interval. Recall is document retrieval against these labels, not answer accuracy. Online integrity remains UNKNOWN.

High recall can coexist with source-quality defects

High recall can coexist with source-quality defects. Exact values in the following table.
Observed count — exact aggregate values
Condition Observed count
Capture unavailable 4.000000
Empty processed text 1.000000
Malformed processed text 1.000000
Duplicate content groups 1.000000
Stale timestamp 1.000000
Missing timestamp 1.000000
Required content absent 1.000000
Same 20-document, 20-query local workload, one CPU / 768 MiB observer; Recall@3 remained 1.00. Counts are independent artifact/policy observations, not a composite quality score. Documents can have multiple issues; the duplicate bar counts groups, not documents. Source coverage was 16/20; missing artifacts remain unavailable evidence. One controlled condition, no inference about production defect rates. Indexed content was unchanged; this does not show that corrupt content is harmless.

Detection parity across three configured workflows

Detection parity across three configured workflows. Exact values in the following table.
Actionable incident observations detected — exact aggregate values
Workflow Detected / expected False attention Complete runs Incomplete controls
Native + IR 285/285 0 27 6
Phoenix + Pandera 285/285 0 27 6
Verifold 285/285 0 27 6
BEIR SciFact: 5,183 corpus documents, 100 development queries, three synthetic fault seeds. Each method: 27 complete runs plus six incomplete/unavailable controls. 285 repeats across individual/mixed conditions are not 285 independent bugs. Zero root causes proved. Same upstream metadata and custom policies required. Actual Phoenix Evals 3.9.4 deterministic code evaluators + Pandera 0.35.0; no hosted competitor or LLM judge tested. Phoenix is Elastic-2.0 source available. Client bound: one CPU, 1,536 MiB; server compute unmeasured. Research-only enlarged scan bounds were used; the developer beta remains limited to 1,000 source rows / 4,000 points. No superiority claim.

Targeted repair costs less than rebuilding this index

Targeted repair costs less than rebuilding this index. Exact values in the following table.
Mean owner-process CPU seconds — exact aggregate values
Strategy Owner CPU s Workflow CPU s Re-embeddings Final nDCG@10
Targeted repair 6.541165 15.929565 40 0.655160928
Full reindex 433.262564 442.229920 5183 0.655160928
5,183 SciFact documents, all 300 final queries, three fault seeds; same per-query final retrieval metrics in every seed. 40 re-embeddings versus 5,183. Both remove ten ineligible copies. CPU = measured user + system process time, not wall time, energy or money. Full workflow adds diagnosis, policy, isolation and grading CPU. One CPU / 1,536 MiB client; server/build/download compute excluded. This compares repair strategies, not Verifold against competitors. Native and Pandera identified the same affected records.

Advisor benefit over competent controls is inconclusive

Advisor benefit over competent controls is inconclusive. Exact values in the following table.
Advisor minus comparator nDCG@10 — exact aggregate values
Condition Advisor minus comparator nDCG@10 Conditional 95% interval
Severity / age 0.000374 [-0.000194, 0.001347]
Query usage 0.002585 [-0.000064, 0.007640]
Random 41 -0.000191 [-0.001352, 0.000796]
Random 42 0.002506 [0.000376, 0.005229]
Random 43 0.004537 [0.000378, 0.010755]
Client bound: one CPU / 1,536 MiB. Budget: 15 optional re-embeddings plus mandatory removals for every active method. All 300 SciFact test queries, three fault seeds averaged per query, 201 shared-document/incident clusters. Dots: paired mean differences; lines: 2,000-resample conditional cluster-bootstrap 95% intervals. These are exploratory, not population or multiple-comparison-adjusted claims. Advisor mean nDCG@10: 0.654548; severity/age: 0.654174; query usage: 0.651963; random controls: 0.654739 / 0.652042 / 0.650011. Overlap with zero does not prove equivalence. Optional selection uses development judgments only; final grading follows frozen plans. No automatic repair in the product.

The small training-selection test did not establish improvement

The small training-selection test did not establish improvement. Exact values in the following table.
Mean nDCG@10 — exact aggregate values
Method nDCG@10 Seed range Queries improved / regressed
Unchanged 0.655160928 — —
random 0.655177543 0.655112812–0.655258891 1 / 1
Dedup / stratified 0.654213994 0.654149262–0.654295341 1 / 2
TracIn adaptation 0.654197378 0.654197378–0.654197378 0 / 1
Only 384 query-coordinate weights were trained on frozen MiniLM embeddings. No encoder or LLM was trained. 512 eligible examples, 128 selected, equal initialization and 2,048 fixed-dimensional example updates; three seeds per method, all 300 SciFact test queries. Seed ranges are not confidence intervals. The TracIn adaptation improved zero distinct queries and regressed one; all long-claim metrics were unchanged. Preparation used 6.74 process CPU seconds; nine selections/trainings together used 0.50, with grading charged separately. One CPU / 1,536 MiB client; no training-token or frontier-model benefit established. Public pretraining contamination remains unknown.

THE CORRECTNESS CONTRACT

Coverage is not certainty.

Reading every declared item does not turn observations from separate systems into an atomic snapshot.

UNKNOWN online.

Online and stage reports retain global UNKNOWN—even with matching observations, repeated mismatches or upstream progress claims.

PASS has prerequisites.

The offline metadata mode requires complete scoped evidence, stopped writers and retained deletion history under its explicit consistency contract.

No automatic repair.

Verifold reads and reports. Source authority, complete inventories, retained deletions, expected chunks and permitted access are adoption requirements.

The wider source-to-retrieval platform is a product direction. The implemented connectors and experiments above define the current supported scope.