Adversarial Validation of Computational Biology Claims

Computational-biology claims require adversarial validation: an intentional effort to challenge citations, statistical interpretations, feature construction, confounding control, null models, sample-size statements, and data provenance rather than only checking whether an analysis runs. ₁ This practice complements evidence-triangulation-for-functional-annotation, provenance-aware-resource-discovery, and cross-tenant-data-bridging by testing whether apparently supported results remain valid under independent scrutiny. ₁

What adversarial review adds

Standard review and adversarial review identified different classes of problems in the reviewed projects. ₁ Standard review described one project as exceptionally sophisticated with no critical issues and recommended proceeding with the existing Tier-A candidate list, whereas adversarial review of the same files found 5 critical and 6 important issues. ₁ The adversarial findings included feature leakage in within-cluster differential-abundance testing — samples were clustered on taxon abundances and the same taxa were then tested within cluster — hard-coded “RESOLVED” verdict logic that treated non-significant results as positive evidence, zero confounder adjustment, a Jaccard 0.14 reported as supporting H2b without any null distribution, an ecotype framework with 48.9 % cross-method agreement and no external replication, and sample-size overclaims. ₁ This contrast supports treating adversarial review as a distinct quality-assurance layer rather than as a more skeptical wording pass. ₁

The phage-targeting project’s own record is the inside account of that same contrast rather than an independent second case: two independent routine review runs on the pre-rigor-repair NB04 state concluded “no critical issues”, while an adversarial reviewer — a general-purpose agent given an explicit “find flaws” framing — caught 5 critical + 6 important issues, all empirically confirmed by the follow-up notebooks NB04b and NB04c. ₂ That project’s narrative also reports 23 notebooks (Pillars 1–5) and two adversarial reviews catching 9 critical + 16 important methodological issues, so the adversarial yield was not confined to a single round. ₂ The central digest attributes the 5-critical/6-important contrast to this same project, so the two passages refine one account rather than replicating it across projects. _1,2

The division of labour is specific rather than a matter of reviewer diligence: the standard reviewer caught surface flaws such as missing sections, undocumented dependencies, broken figures and simple statistical errors, and was systematically weak at structural and inferential issues — selection-on-outcome confounding, missing null distributions, post-hoc choice of decision rules, and narrative-versus-evidence mismatch. ₁ This refines the contrast above: what the adversarial pass supplies is an explicit framing that invokes those checks, not more scepticism applied to the same checks. ₁

Across 9 adversarial review rounds, two recurring citation failures were PMID hijacking and hallucinated author lists. ₁ PMID hijacking occurred when real PMIDs were associated with unrelated papers, while hallucinated author lists created unsupported bibliographic details. ₁ Load-bearing citations therefore require verification of the author, title, journal, PMID, and DOI through PubMed rather than relying only on DOI resolution. ₁

The fabrications in one project’s record were the reviewer’s rather than the author’s, which makes the review stream itself an object of audit. ₃ The functional-atlas project integrated 7 adversarial reviews (REVIEW_1 plan-stage through REVIEW_7 Phase 3 closure) and documented three reviewer-side fabrications — a Mendoza 2020 PMID-hijack at REVIEW_5, a Koech 2025 fabrication at REVIEW_7 and a Burch 2023 PMID error at REVIEW_7 — after which a verify-before-acting protocol was established and reinforced. ₃ REVIEW_4, REVIEW_5 and REVIEW_7 each carried fabricated citations verified via PubMed, while REVIEW_8 had 0 fabrications and 7/7 DOIs auto-verified. ₃ The per-round tally records one Alm 2006 hallucination (a false claim) at round 4, one Mendoza 2020 PMID-hijack at round 5, one Koech 2025 fabrication plus one Burch 2023 PMID error at round 7, and 7/7 citations verified by the auto-verifier with zero fabrications recorded at round 8. ₃ This refines the PMID-hijacking failure class above: the same failure occurs on the reviewing side, so an adversarial finding is load-bearing only after its own citations resolve. _1,3

The hijack is documented in detail. The reviewer advanced Mendoza et al. 2020, “Hologenome evolution: mathematical model with horizontal gene transfer” (Microbiome 8:1, doi:10.1186/s40168-020-00825-5, PMID:32160912), as supporting evidence, and the project subsequently identified that citation as fabricated. ₃ PMID 32160912 actually points to Rezaeipandari et al. (2020), a Persian-language quality-of-life questionnaire study in Health and Quality of Life Outcomes 18(1):67, doi:10.1186/s12955-020-01316-0, completely unrelated to horizontal gene transfer (HGT), and PubMed searches for the purported Mendoza HGT topic return 0 results. ₃ The fabricated reference carried a specific quantitative claim — that regulatory genes show 50-fold lower transfer rates than metabolic genes — which was the cornerstone of the reviewer’s finding I1 and of the H1 hypothesis-vetting “partially supported” verdict, so an invented source was doing inferential work inside the review. ₃ These verifications are as reported by that project and are not independently rechecked here. ₃

Adversarial findings can also be plainly false or overcalled, so they need the verification discipline they demand of the claims they attack. ₃ REVIEW_4’s C1 asserted that Alm 2006 was not cited, a claim checkable by reading references.md line 9. ₃ The project records Alm 2006 as the first reference in references.md (line 9, “Methodological anchor” section), referenced ≥30 times across DESIGN_NOTES.md, with a dedicated 460-line comparison memo linked from the README. ₃ A second critique, “no concrete multiple-testing strategy”, is recorded as partially overcalled: a hierarchical Tier-1 / Tier-2 / Tier-3 strategy existed in the plan, while the implementation details — the BH-FDR (Benjamini–Hochberg false discovery rate) variant and the effective-N for KEGG-BRITE × GTDB-family clusters — were still TBD and a genuine later deliverable. ₃

Round dispositions show the mix. REVIEW_5 raised 4 critical + 6 important + 3 suggested issues against the v1.8 milestone, of which one important finding traced to the fabricated citation, three critical findings restated concerns already addressed, two misframed defensible methodology, and two genuine new concerns — a small-n caveat and M22 biological-validation anchors — drove plan-level commitments. ₃ REVIEW_8 raised 3 critical + 4 important + 2 suggested issues against the v3.0 Final Synthesis, with its citation auto-verifier resolving 7/7 DOIs and no reviewer fabrications detected that round; its C9 numerical claims (n=2,350, d=0.08 at PSII genus rank) were checked as accurate against the project’s own phase-gate decision data, C7 was already addressed, and C8 was judged partially valid but to conflate methodology-revision-as-transparency with multiple-hypothesis testing. ₃

Adversarial review is also productive at plan scope, before the plan’s downstream notebooks are executed. ₁ Adversarial review of one plan revision found 4 critical and 10 important issues, including feature leakage in an N14 dual-basis design whose same-axis ecotype basis is leakage-poisoned, a “4-substudy meta-viable” claim inherited without re-verification that was actually 3+1, thresholds without null distributions, a “butyrate-producer ↔ pathobiont” label in which 4 of 6 named anchors were not butyrate producers, a strain-adaptation test that collapses to species level by construction, and an H3e sample of n=67 across 3 sites rather than the claimed 130-subject HMP2 cohort. ₁ Every category of issue found at notebook scope in one pillar appeared at plan scope in this revision, which supports selection-on-outcome-leakage as a design-time rather than analysis-time concern. ₁

The corpus therefore recommends that any plan revision whose downstream notebooks will be load-bearing for clinical, experimental or external claims be reviewed by both a standard reviewer and a paired adversarial reviewer, since the adversarial pass is what catches feature leakage, missing nulls, hard-coded verdict logic, sample-size overclaims and label-versus-data mismatches. ₁ The economic argument is direct: the same class of repair took 7 notebooks when caught at notebook scope but a single plan-edit pass when caught at plan-revision scope. ₁

Recurring failure classes

Four failure classes recur across the reviewed work. ₁ Effect-size thresholds were reported without null distributions — a Jaccard of 0.14 in one notebook and |ρ| > 0.4 in one plan. ₁ Substudy and cohort sample-size claims did not match the data on inspection. ₁ Hypothesis labels did not match the operationalized test: one notebook recorded “H2c RESOLVED” while treating a non-significant result as positive evidence, and one plan’s “butyrate-producer” anchors were mostly non-butyrate-producers. ₁ Falsifiability rules were loose enough to pass under random data, as with a rule of ”≤ 5 categories of 7” that passes on random allocation. ₁

The common thread is pre-registration discipline: what the actual sample size is given the actual data, what null distribution any positive claim is being made against, and what each operational threshold means given that null. ₁ Standard review treated these as advanced questions while adversarial review treated them as entry-level diligence. ₁

The cost of leakage and unadjusted confounding is quantifiable, and repair does not always overturn the biology. ₁ One pillar reported a 33-species within-ecotype Tier-A list (18 E1, 15 E3) with an H2c C. scindens paradox marked “RESOLVED by stratification”, and two rigor-repair notebooks then applied three evidence filters to every candidate. ₁ The H2b ecotype-divergence hypothesis nonetheless survived strongly under a permutation null, with p = 0.000 for an observed Jaccard 0.104 against a null mean of 0.785 ± 0.054, so the original effect-size framing was the wrong statistic rather than the wrong conclusion. ₁ The phage-targeting project records the same repair in its own terms: the permutation null for Jaccard(top-30 E1, top-30 E3) under randomized ecotype labels yields mean 0.785 ± 0.054 against an observed Jaccard = 0.104, empirical p = 0.000 over 200 permutations, which retracts the earlier “Jaccard = 0.14 supports H2b” framing while leaving the ecotype-divergence conclusion standing — the conclusion survived, the statistic did not. ₂ The companion H2c claim did not survive: under a confound-free within-IBD-substudy CD-vs-nonIBD meta-analysis C. scindens is genuinely CD↑ with pooled CLR-Δ (centered-log-ratio difference) = +1.18, FDR = 1e-8 and 4/4 sign concordance across sub-studies; the earlier within-ecotype “n.s.” call was an artifact of feature leakage — clustering samples on taxon abundances and then testing the same taxa within cluster — and under leave-one-species-out refit C. scindens is CD↑ in both E1 and E3, so the “paradox RESOLVED by stratification” verdict is contradicted and retracted rather than confirmed. ₂

The retraction reached the candidate list as well as the statistic. The 33-species NB04 within-ecotype Tier-A list (18 E1, 15 E3) is retracted because the within-ecotype differential abundance that produced it was substantially driven by feature leakage — held-out-species sensitivity Jaccard E1 = 0.230 and E3 = 0.064 against a > 0.5 leakage-bounded threshold — and because, tested against an independent confound-free CD-vs-control contrast, 14 of 18 E1 candidates had negative within-substudy effects, making them ecotype markers rather than CD drivers. ₂

The project’s own stopping-rule scorecard records which pre-registered criteria the repaired analysis failed: the feature-leakage bound required a held-out-species Jaccard > 0.5 and returned 0.230 for E1 and 0.064 for E3, both failing; ecotype-specific Tier-A required ≥ 3 and returned 51 for E1 and 40 (single-study) for E3, both passing; engraftment pathobionts under the confound-free contrast required ≥ 3 of 6 and returned 5/5 tested at cohort level, passing; and ecotype-framework internal stability required a bootstrap adjusted Rand index (ARI — agreement between two clusterings corrected for chance) > 0.30 and returned 0.16, failing. ₂ This supports ecotype-clustering-validity: the framework that generated the retracted strata did not meet its own stability bound. ₂

The confound-free replacement is where the project’s surviving signal lives, and it refines rather than erases the retracted work: across the four IBD sub-studies (242 CD / 369 nonIBD pooled), the curated battery reproduces the canonical CD signature of pathobionts up and protective commensals down, with pooled CLR-Δ, FDR and sign concordance of +5.13, ~0 and 4/4 for Mediterraneibacter gnavus; +2.30, 4e-9 and 4/4 for Eggerthella lenta; +1.43, 2e-4 and 3/4 for Escherichia coli; +1.18, 1e-8 and 4/4 for Clostridium scindens; +1.09, 3e-6 and 4/4 for Enterocloster bolteae; +0.92, 5e-4 and 3/4 for Hungatella hathewayi; +0.07, n.s. and 3/4 for Bilophila wadsworthia; −1.01, 4e-3 and 3/4 for Lachnospira eligens; −1.14, 2e-3 and 4/4 for Roseburia intestinalis; −1.30, 3e-3 and 3/4 for Akkermansia muciniphila; −1.67, 9e-8 and 4/4 for Faecalibacterium prausnitzii; −1.77, 4e-7 and 4/4 for Roseburia hominis; and −3.09, 4e-15 and 4/4 for Coprococcus eutactus. ₂ The battery is labelled 14-species while 13 species rows are displayed, so the panel size is unresolved and is not recomputed here; this bears on study-batch-confounding-of-environmental-associations, since the within-substudy design is what removes the cohort confounding the earlier contrast carried. ₂

A repair can also fall short of its own plan, and this one records the shortfall rather than claiming the plan. The plan called for ≥ 2 / 3 of {ANCOM-BC, MaAsLin2, LinDA} differential-abundance methods to agree; LinDA was implemented in pure Python to avoid the R/rpy2 dependency, and the bootstrap-confidence-interval and within-substudy-meta-analysis outputs are offered as additional independent evidence streams but are explicitly not drop-in ANCOM-BC / MaAsLin2 substitutes, with a full three-method R-native consensus left as a publication-grade follow-up. ₂ That independence claim is qualified in the same report, and the tension is recorded rather than resolved: CLR-MW (NB04) and LinDA (NB04c) operate on the same ecotype-defined subsamples and therefore share the selection-on-outcome bias, only the within-substudy CD-vs-nonIBD contrast is described as an independent evidence source, and the NB04d Tier-A gating requires within-substudy concordance precisely because bootstrap + LinDA are not independent evidence streams. ₂

A failed verdict can itself be a schema artefact rather than a refutation. ₁ A v1.7 “FAIL — degenerate” verdict was technically correct given the v1.7 schema and was reversed by the v1.8 schema, and the diagnostic notebooks had correctly noted the structural underpoweredness; the operational lesson recorded is that the verdict should have been read as a flag to revisit the schema, not as a true refutation. ₁ This ties adversarial validation to ontology-and-category-schema-sensitivity. ₁

The phage-targeting project’s three-clause v1.7 H3a test is recorded in the same spirit: clauses a and c PASS while clause b is FAIL-degenerate, which is the degenerate verdict the schema revision later reversed. ₂

Closing a falsifiability framework is not the same as confirming it. The same project reports the H3 framework as fully closed with 5 SUPPORTED + 1 PARTIALLY SUPPORTED + 2 PARTIAL (single-cohort caveat) + 1 NOT SUPPORTED (with methodology lesson), and its verdict-score line restates 5 SUPPORTED + 1 PARTIALLY SUPPORTED + 2 PARTIAL + 1 NOT SUPPORTED; the categories are retained as stated rather than collapsed into a pass rate. ₂ The formal H3c genomic-mechanism verdict is PARTIALLY SUPPORTED, which qualifies the enrichment results that rest on it. ₂

Superseded outputs retained for audit

A retraction leaves artefacts, and this project keeps them instead of deleting them: the 33-row data/nb04_tier_a_candidates.tsv original within-ecotype Tier-A table is marked retracted and kept for audit, and the original within-ecotype CD-vs-HC CLR differential-abundance output is marked superseded and kept for audit with NB04 claims 1 and 2 retracted. ₂

The superseded figures are retained with their contents described rather than withdrawn silently: NB04_h2c_paradox_resolution.png is the NB04 heatmap of 14 curated species × pooled raw / pooled CLR / E1 CLR / E3 CLR, NB04_top_tier_a_per_ecotype.png is the union of the top-15 NB04 within-ecotype Tier-A, and the legacy NB01 K = 8 fit figures are preserved for methodology audit. ₂

The replacement analysis carries its own figure: NB07a_pathway_DA_h3a.png is the NB07a §a–c three-clause falsifiability dashboard, combining a per-pathway CLR-Δ histogram, a permutation null and a top pathway-pathobiont attribution scatter. ₂

Statistical and interpretive challenges

Methodology revisions M1–M26 were not automatically one multiple-testing family. ₁ The reviewed project had 4 pre-registered hypotheses, for which Bonferroni correction used α=0.0125, and all 4 hypotheses survived that correction. ₁ Only M14, M21, and M23 were genuine post-hoc metric corrections. ₁ The appropriate rule is to correct actual hypothesis-test families, not transparent documentation of methodological changes. ₁

The functional-atlas project’s own accounting refines that rule while also showing its limits. ₃ It documents 25 methodology revisions (M1–M25), each with rationale and triggering observation in its plan revision history, plus three pre-registration omissions surfaced and corrected as project-discipline lessons: M2 dosage biology, M12 absolute-zero criterion and M14 misreading-of-Alm-2006; this count differs from the M1–M26 range recorded in the central digest, and neither figure is adjusted here. _3,1 The report disputes treating M1–M25 as 25 alternative hypothesis tests on the same data, classifying most as pre-registration corrections, substrate audits, biological-control additions and metric corrections discovered via diagnostic tests — M1 positive controls AMR/CRISPR-Cas, M2 dosage biology, M12 absolute-zero criterion explicitly, M14 replacing parent-rank dispersion with Sankoff parsimony (a tree-aware minimum-change count of gain and loss events), M21 strict-class housekeeping — that is, transparency about what was done. ₃ The same report nevertheless concedes that the cumulative revision count indicates pre-registration was insufficiently calibrated against substrate-resolution behaviour, even while arguing that most revisions originated in genuine biology engagement (M2 dosage cost, M14 Alm 2006 close-reading, M17 Pseudomonadota AMR bias) rather than data accommodation. ₃

Adversarial validation should explicitly test whether non-significant results are being described as positive findings, whether covariates have been adjusted, whether a null distribution exists for an enrichment or association claim, and whether the reported sample size matches the independent observational units. ₁ These checks are especially relevant to phenotype-database-coverage-bias, phylogenetic-confounding-of-pangenome-associations, and fitness-matched-null-models, where dependence structure and null-model choice can change the interpretation of an association. ₁

Pre-registered thresholds and criteria under audit

Pre-registered decision rules are themselves auditable objects, and projects in this corpus found their own rules uninformative on re-reading. ₄ In the Caulobacter lipid-A-loss project the pre-registered ≥10% threshold sits below the background rate, so threshold-passing is by itself uninformative about specific mechanistic importance. ₄ The report states that this threshold is essentially a presence test rather than an enrichment test, and makes hypergeometric enrichment against background the formal H2 verdict in the revised report. ₄ This supports fitness-matched-null-models: a threshold without a background model is not an enrichment claim. ₄

The same project held a pre-registered criterion to its stated form rather than relaxing it after the fact. ₄ The CtpA BORDERLINE criterion required both 0.05 < pvalue < 0.15 and protein detection; the observed pvalue = 0.048 fails the lower bound and CtpA was not detected in the outer-membrane proteome, so BORDERLINE was not met. ₄ A cumulative 4599-vs-4580 contrast is significant (logFC +0.76, FDR — false discovery rate, the expected proportion of false positives among called hits — 0.035) but conflates Δfur + ΔsspB and Δlpxc effects, whereas the pre-registered 4599-vs-4584 contrast was designed to isolate the Δlpxc-specific response; the cumulative result is therefore reported as a supplemental observation consistent with CtpA being Fur-regulated and not as evidence for lipid-A-loss-driven CtpA induction. ₄

Text-matching construction of curated gene sets admits false positives that survive into pre-registered tests. ₄ CCNA_00565 (γ-glutamyltranspeptidase, a glutathione enzyme) and CCNA_01833 (glucosylceramidase, a ceramide enzyme) entered a pre-curated peptidoglycan-remodeling set via description-text matching; they are present in the union of significant hits but do not change the H4 verdict, which remained well above its ≥3 threshold, and were flagged so downstream interpretation does not lean on them. ₄

A pre-registered criterion can also fail because it encoded an absolute null where the biology predicted a signed effect. ₃ The functional-atlas project’s v2 negative-control criterion specified “producer mean within ±1σ of zero” for housekeeping classes (ribosomal, tRNA-synthetase, RNAP core), but its Phase 1A pilot showed all three negative controls at producer z ≈ −0.15 with confidence intervals entirely below zero, and the criterion was revised post-hoc to “CI upper bound ≤ 0.5”. ₃ The adversarial reviewer’s objection is recorded as fair: revising the negative-control criterion (“near zero” → ”≤ 0 with CI not strongly positive”) after seeing data weakens pre-registration discipline. ₃ The project’s stated defence is that the dosage biology — dosage-constrained genes have fewer paralogs than typical genes at matched prevalence (Andersson 2009; Bratlie et al 2010) — was correctly anticipated during plan v2 design but its quantitative consequence (negative producer z, not zero) was never encoded into the criterion, so M2 corrects a pre-registration omission rather than redefining a target, and the corrected expectation should itself be pre-registered in a later phase. ₃

The same project’s positive controls failed an absolute threshold that the realised null did not support. ₃ Plan v2 defined “Innovator-Exchange” as both producer and consumer 95 % CI lower bound > 0, and all positive controls (β-lactamase, CRISPR-Cas, AMR, two-component-system (TCS) histidine kinases) sat at strongly negative consumer z and failed it, while a post-gate diagnostic found positive HGT classes +0.6–1.2 σ less clumped than housekeeping at family rank (p << 10⁻⁷) — the absolute-zero threshold was over-stringent at UniRef50 resolution, where the null distribution is itself shifted. ₃ M12 therefore reformulated Innovator-Exchange as a relative-threshold metric (”≥ X σ less clumped than housekeeping baseline at the rank’s parent”) and made the HGT-class-versus-housekeeping delta a primary atlas metric alongside absolute consumer z; at this stage the threshold X was still to be calibrated empirically from Phase 2 data rather than already fixed. ₃ The project also records that its own gate-decision wording, “HIGH 1 known-HGT controls fail”, overstated the result — the controls fail the absolute Innovator-Exchange threshold but pass the relative-discrimination diagnostic — and that amplification by KO aggregation remains an expectation rather than a measured result. ₃ This supports fitness-matched-null-models from the opposite direction to the lipid-A case: a threshold set against zero rather than against the realised null can fail on true positives. ₃

A criterion relaxed after seeing data must not then adjudicate the hypotheses it was relaxed around, and this project draws that line explicitly. ₃ The Bacteroidota polysaccharide-utilization-loci (PUL) hypothesis remains falsified at the pre-registered absolute-zero criterion, with M12’s relative-threshold framing applying only to control-class methodology validation — whether the methodology distinguishes HGT-active classes from housekeeping baselines — and not to hypothesis adjudication. ₃ Plan v2.7 separates the two uses: the four pre-registered hypotheses (Bacteroidota PUL, Mycobacteriota mycolic-acid, Cyanobacteria PSII, Alm 2006 TCS reproduction) retain absolute-criterion adjudication, while M12 governs methodology quality control only. ₃ This refines horizontal-gene-transfer-driven-innovation: the falsified PUL verdict stands on the criterion as written. ₃

Null-model construction can be circular in the same way feature selection can. ₃ In this project the natural_expansion class is selected for paralog count ≥ 3 and then tested against a null designed to detect paralog expansion, so it cannot validate that null. ₃ Plan v2.7 binds M20, an independent paralog holdout for producer-null validation at Phase 2 sourced from curation independent of the natural_expansion construction (Pfam clans with documented paralog families per Treangen & Rocha 2011; KO orthogroups with cross-organism duplicates per the Csurös 2010 Count framework) — a plan commitment, not a reported completed result. ₃ This supports selection-on-outcome-leakage at the level of the null model rather than the feature matrix. ₃

An unmatched background can inflate a particular effect estimate, and a missing null distribution leaves a positive claim unsupported rather than necessarily manufactured. ₂ The phage-targeting project records that its NB08a Tier-A “background” comparator is the full biosynthetic-gene-cluster (BGC) catalog — 10,060 BGCs across all species — rather than a matched comparator, so the Fisher odds ratio of 44× for iron_siderophore is partly inflated by the catalog-wide rarity of iron MIBiG matches (51 / 9,774 = 0.5 %); a matched-niche pathobiont set (gut Proteobacteria + CD-associated Firmicutes) at similar genome-assembly depth would be more conservative, the 44× should not be treated as a precise effect size, and only the qualitative finding — E. coli alone among actionable Tier-A carries the iron-siderophore signature — is held robust across choice of comparator. ₂ This supports fitness-matched-null-models from a third direction: the comparator set, not only the permutation scheme, is part of the null. ₂

Marker identity and withdrawn narratives

One consequential error class documented here is a marker that passes a similarity threshold while encoding the wrong biology. ₅ An earlier clay project used K07811, K17324 and K17323 as iron-reduction markers, but those KEGG Orthology (KO) identifiers are TMAO reductase, glycerol ABC transport and glycerol permease respectively, and KEGG has no canonical KO for the Geobacter omcS / Shewanella mtr operon multi-heme outer-surface cytochromes. ₅ The follow-up report records the thresholds involved — K07811 (TMAO reductase, threshold 1336.87), K17324 (glycerol ABC ATP-binding, threshold 363.93), K17323 (glycerol ABC permease, threshold 318.27) — attributes the earlier “Mitzscherling rock-attached signature in shallow clay” framing to mismatched gene IDs combined with a standard eggNOG-mapper threshold that filters on sequence similarity rather than KO biological identity, and reports that after correction the iron-reduction comparison loses statistical significance entirely. ₅ The clay project itself withdraws its iron-reduction account of a shallow-clay, rock-attached pattern. ₆ This supports functional-marker-validation: marker identity must be validated independently of the score that selected it. _5,6

Allowlist errors behave the same way at compound scale, and fixing the rule is not the same as fixing the outputs. ₇ An ENIGMA carbon census corrected its NB03 carbon allowlist to exclude R02107, which makes the carbon-callable set effectively 8, while the committed tables still list xanthine because they predate that fix and were not regenerated. ₇

Anchor papers, effect-size units, and diagnostic discipline

A foundational citation can be cited accurately and still be misread, with the misreading propagating into infrastructure. ₃ A close-reading of Alm 2006 revealed three load-bearing misreadings in the functional-atlas project: the four-quadrant Open/Broker/Sink/Closed Innovator framework was the project’s own construction and was never named in that paper, which reported a correlation r = 0.74 between HPK (histidine protein kinase) count and recent-LSE (lineage-specific expansion) fraction; the original worked at the single-domain level (≈ UniRef50, not “family”); and it used phylogenetic-tree-aware reconciliation, not parent-rank dispersion. ₃ The substrate-hierarchy claim therefore needed reformulation from “UniRef50 is too narrow, aggregate to KO” to “we used the wrong metric on the right substrate”, M14 reframed the project’s relationship to that paper from “generalizing” to “Alm-2006-inspired”, and M15 promoted Sankoff parsimony to mandatory. ₃ The recorded lesson is that anchor papers cited as foundational must be close-read for exact methodology before infrastructure is built on extrapolations, as a Phase-0 deliverable. ₃

Overstatement can come from switching units rather than from inventing numbers. ₃ Sankoff discrimination at UniRef50 gave Cohen’s d = 0.15 — a standardized mean difference between classes — well below the d ≥ 0.3 the project pre-registered for atlas-grade discrimination, and the earlier “+0.6 to +1.2 σ less clumped than housekeeping” framing was expressed in parent-rank z-units rather than Cohen’s d and overstated effect magnitude. ₃ The softened reading holds that HGT signal at UniRef50 is real but small and that Phase 2 KO aggregation is expected to amplify it, which is an empirical claim Phase 2 must demonstrate rather than a substrate-hierarchy guarantee. ₃ The Phase 1B verdict was recorded as PASS_REFRAMED + qualified: the methodology was not broken, but the substrate-hierarchy framing became an empirical bet carrying a sharp falsifiable test. ₃

The corrective this project reports is cheap and local: when a result deviates from pre-registered expectations, run a focused diagnostic before invoking substrate, methodology or “we need more data” explanations. ₃ Three such diagnostics — a Phase 1A power analysis, a Phase 1B discrimination diagnostic and a Phase 1B Sankoff diagnostic — collectively took ~3 hours to write, ran in seconds-to-minutes, and dispositively distinguished metric and criterion problems from methodology failures. ₃ A diagnostic can nevertheless drift from its own plan: Sankoff parsimony was scored on the GTDB-r214 species tree pruned to 18,989 Phase 1B leaves for all 748 panel KOs in 11 seconds, with pairwise Cohen’s d and cluster-bootstrap confidence intervals at B = 200 KO-level resampling within class, a different granularity from the Pfam-family-level bootstrap that M19 specified. ₃

Corrected metrics and circular validation

Effect sizes can be wrong by orders of magnitude through the choice of denominator alone. ₈ The plant-microbiome project’s NB06 reported Cohen’s d = −7.54 for co-occurring versus random genus pairs, which the project itself flags as an implausibly large effect, and traced it to dividing the mean difference by the standard deviation of permutation means — which shrinks toward zero by the central limit theorem — instead of by the raw pair-level standard deviation. ₈

Correcting the formula on the same data gives d ≈ −0.43 for the max-aggregated complementarity, and switching aggregation from species-max to species-prevalence further reduces |d| to 0.39, so of the apparent “20× reduction” reported in an early draft the project attributes ~90% to the formula correction and ~10% to the aggregation correction. ₈ The canonical finding that survives is small: co-occurring plant-associated genera are slightly less complementary than random genus pairs (Cohen’s d ≈ −0.4, permutation p < 0.001), the prevalence-weighted re-test plus the formula correction takes the original implausible |d| = 7.54 to |d| ≈ 0.4, and the redundancy direction is called robust at that resolution — which is the GapMind pathway level of 80 pathways, a resolution the report itself flags as possibly too coarse and as able to mask patterns that reaction-level or substrate-specific analysis could reveal. ₈ This refines community-metabolic-interdependence: the redundancy sign stands at pathway-level resolution while its magnitude does not. ₈

A validation metric that cannot fail is not evidence. ₈ An earlier draft of the same project reported a “77.8% relaxed accuracy” by counting the dual-nature class as correct for both beneficial and pathogenic ground truth; the paired adversarial review noted that this rule cannot fail for any species the classifier assigns to dual-nature, so the number is arithmetically 14/18 but methodologically tautological and carries no discriminative information. ₈ The project’s earlier “92.7% genus-level agreement” figure was circular for the same reason — the cohorts are defined from marker presence — and was replaced by a curated panel of 18 model organisms with experimentally confirmed phenotypes (7 beneficial, 7 pathogenic, 4 neutral). ₈

The replacement test returns a null for the categorical label and only a weak signal for the continuous one. ₈ All 14 of the beneficial plus pathogenic ground-truth species in that panel were assigned to the dual-nature class, so the categorical cohort label is uninformative at species level and cannot discriminate beneficial from pathogenic ground truth; the discriminating quantity is instead the continuous pathogenicity ratio n_pathogen / (n_pgp + n_pathogen), lower in known beneficial species (median 0.50) than in known pathogens (median 0.60) at Mann-Whitney U = 9, p = 0.027 on N = 7 vs 7 — a small effect at small sample size. ₈ This supports functional-marker-validation: marker-derived labels need ground truth independent of the markers. ₈

A null result can be an identifier artefact rather than biology. ₈ The same project’s NB12 null for H7 — 5 species tested, 0/5 significant — was caused by a genome-ID format mismatch between phylogenetic_tree_distance_pairs (bare NCBI accessions) and genome_environment.csv (GTDB-prefixed IDs), which left the is_plant_associated column entirely NaN after the merge. ₈ This supports taxonomic-nomenclature-reconciliation: a join-key mismatch and a true negative are indistinguishable in the output. ₈

After prepending GB_/RS_ prefixes to the bare accessions, 599 plant-associated genomes were recovered across the 5 species, and the full 18-species scan — limited to species with phylogenetic-tree distance data — found 5/17 testable species passing Bonferroni-corrected Fisher’s exact test, with P. avellanae (Fisher p = 7.3×10⁻⁴), X. vasicola (Fisher p = 1.6×10⁻⁴) and X. campestris (Fisher p = 1.0×10⁻³) the three most robust. ₈ That coverage is partial rather than unrestricted: 47 candidate species had no phylo-tree coverage, and the report states that the true H7 effect across all 65 candidate species cannot be estimated until the GTDB tree is more completely populated. ₈ The earlier five-species draft also flagged P. amygdali on chi² alone, but that result fails Cochran’s rule and Fisher’s 2×2 collapse and is not carried forward; this bears on genome-wide-versus-locus-specific-ecological-adaptation, where which species pass determines whether adaptation reads as locus-specific. ₈

Two further self-corrections are recorded in the same project. An earlier per-orthologous-group test used a prevalence-based Bernoulli simulation that the paired adversarial review flagged as circular by construction; it was replaced with the real per-species orthologous-group (OG — a cluster of genes inferred to descend from a single ancestral gene) matrix from NB03 and re-run. ₈ The overstated “novel OGs” terminology was reframed to “enriched gene families”. ₈ Three fixable adversarial-review issues — the tautological genus-level validation, the original zero-hit Pfam query (documented as a limitation) and the NB12 subclade genome-ID mismatch — were resolved in notebooks NB13–NB15, and four statistical controls that the research plan had specified but Phase 1 had not executed were also completed. ₈

Silent failures at code level

Adversarial validation has to reach the code, because several recorded failures return wrong numbers with no error message. ₉ When merge(..., how='left') is used to create a boolean flag column, unmatched rows become NaN, and calling .fillna(False) converts the column to object dtype rather than boolean, so the ~ bitwise-NOT operator produces silently wrong results instead of boolean negation. ₉ In one case this reported an orphan essential gene count of 41,059 — the total number of essentials — instead of 7,084 actual orphans, with no error raised. ₉

A sensitivity analysis can be void by construction. Setting feats_relaxed = feats_strict.copy() in NB02 produced numerically identical strict and relaxed downstream outputs, which invalidates any mapping-mode sensitivity conclusion because the two modes are no longer independently constructed; the recorded fix is to compute strict and relaxed features independently, for example by precomputing clade-level annotation counts once on the union of clades and aggregating those counts separately per mode. ₉

Multiple-testing correction can silently miss its own targets: a pass restricted to columns ending in _p skips p-value columns named like adj_cov_p_contamination or adj_frac_p_contamination, so p-value columns should be detected by substring pattern — for example columns containing _p while excluding derived q-value columns — or by an explicit p-column allowlist before BH-FDR (Benjamini–Hochberg false discovery rate) correction is applied. ₉

Variable reuse corrupts sequential tests: running Fisher exact tests in sequence while reusing generic names such as odds, p = stats.fisher_exact(table) lets the second test overwrite the first, after which a downstream summary cell prints the wrong values under the wrong labels. ₉

Schema misreadings produce confidently wrong counts. An initial analysis counted exactly 80 pathways for every species with 0% present because it filtered on score_category = 'present', a value that does not exist; the GapMind resource has exactly 80 pathways total and each genome-pathway pair has multiple rows, one per step or component of the pathway. ₉ This supports computational-pathway-prediction-validation: a pathway-completeness number is only as good as the row semantics it aggregates. ₉

The audit trail itself fails in a third way. When analysis is prototyped as Python scripts for debugging speed and the notebooks are then committed with empty output cells, the recorded results that synthesis extraction, reviewer claim-verification and human reading all depend on are absent, and the notebook stops functioning as the primary audit trail and methods documentation. ₉ This connects to analysis-provenance-and-reproducible-outputs. ₉

The constructive counterpart is a unit test designed to fail when the bug is present. One project ships a script that generates synthetic data with known phylogenetic signal (λ = 0.8, β = 0.05), scrambles the row order, and verifies that sorted phylogenetic generalized least-squares (PGLS — regression that models residual covariance from a phylogeny) recovers the correct λ and β within tolerance while unsorted PGLS fails; the test passes. ₁₀ A row-ordering assumption that is only documented is not checked, whereas one encoded as a failing-when-broken test is. ₁₀

Tensions

Several reports state figures that disagree with their own tables; both sides are recorded here rather than reconciled, and no replacement number is computed. _{11,5,4,12,7,2,13,9,8,14,15}

Variance attribution versus the model table. Five-fold phylogenetically blocked cross-validation (repeated model fitting in folds built to keep related genomes together) used n = 3,994 and reported R² of 0.063 for gene count only, 0.163 for phenotype only, 0.354 for taxonomy only, 0.345 for taxonomy + phenotype, and 0.633 for the full model. ₁₁ The report’s narrative instead asserts that the genome’s metal resistance gene content explains more variance than all phenotype features combined, and attributes the full model’s R² = 0.63 with delta R² = +0.28 over taxonomy alone chiefly to adding metal-resistance-cluster count. ₁₁ The gene-count-only value is lower than the phenotype-only value, so the narrative assertion conflicts with the single-predictor values it rests on and must not be carried forward as an established finding. ₁₁

Match-rate arithmetic. The same report states that BacDive species name matching achieves only 38.4% for 5,647/27,702 GTDB species — a percentage inconsistent with that numerator and denominator — and separately gives 38.4% for 37,368 matched strains out of 97,334 BacDive strains, while noting that GCA accession matching could recover additional links but was not implemented. ₁₁ Which quantity the 38.4% describes is unresolved, so coverage claims built on it need re-derivation rather than restatement; this bears on taxonomic-nomenclature-reconciliation. ₁₁

Erratum. The report gives 38.4% for two different denominators. 37,368 of 97,334 matched strains is 38.4%; 5,647 of 27,702 GTDB species, the figure this sentence uses, is about one fifth, roughly half what the percentage suggests. ₁₁

Cohort size. The generated data/bacillota_b_universe.tsv file is listed as having 314 rows while an earlier Results cohort table reports 334 total Bacillota_B genomes, and the report does not reconcile the discrepancy. ₅

Cohort partition. Earlier drafts of the Caulobacter project described an “early cohort of 30 genes” that conflated the unique-to-early genes with the both-phase genes, and separately mis-labeled ChvI as early-only when it belongs to the 10-gene both-phases subset. ₄ The corrected disjoint partition is 20 unique-to-early + 10 both-phase + 49 late, totaling 79 ChvI-induced genes engaged in the data, and the phase-structure verdict is unchanged by the correction. ₄

Theme percentages. The same report gives 6.7% regulator/two-component-system (TCS) genes among the 20 unique-to-early genes, 10.2% envelope/outer-membrane and 12.2% TonB-dependent transporter (TBDT) genes among the 49 late genes, and 0% in early for lipid/membrane and peptidoglycan/cell-wall categories. ₄ The 6.7% figure cannot represent an integer count out of 20, so its denominator is unresolved and the theme-shift description should be read qualitatively rather than as a rate. ₄ The shift is also not statistically discriminated: the Fisher exact test of envelope/transport/regulator enrichment in late versus unique-to-early is not significant (p=0.243), so the report itself calls the “regulator → structural” shift suggestive only. ₄

Negative homology evidence. A method-check scorecard reports LpxA/C/D/B/K as all 0 in PaperBLAST for Caulobacter against 11-18 each in NCBI, and an ~80% PaperBLAST false-negative rate for Caulobacter’s own lipid A biosynthesis genes. ₄ The detailed cross-check table contradicts the scorecard on one gene: PaperBLAST counts of 0, 0, 0, 1 and 0 for LpxA, LpxC, LpxD, LpxB and LpxK against NCBI counts of 11, 15, 15, 18 and 18 make LpxB an under-count rather than a zero-hit false negative, so the all-zero statement conflicts with the table and the exact counts must not be treated as a consistent account. ₄ The report maintains that the headline cross-species claim concerns absence in the comparator species, where NCBI annotation confirms it, so this issue does not threaten that finding; this refines homology-search-negative-evidence, since a text-matching resource’s silence is a measurement with its own error rate. ₄

Sample inventory. The PROTECT study is described as producing 23 structured data tables (30.5M total rows) covering 4,949 isolates from 211 species across 175 patient samples (133 CF, 41 NCFB, 43 subjects, 4 clinical states); the stated CF and NCFB subtotals do not add up to the reported 175 total. ₁₂

Substrate count. The methods sentence describes a carbon source utilization assay testing PA14 and 430 commensal isolates on 20 amino acids plus glucose and lactate, the assay overview states 21 substrates, the selectivity analysis computes selectivity ratios (commensal mean OD / PA14 OD) for “all 20 tested substrates”, and the conclusion reports that among the 22 tested substrates no selective prebiotic exists and that competitive exclusion must instead work through community-level resource depletion rather than individual substrate advantage. ₁₂ All four counts are retained unresolved, and the discrepancy matters for competitive-exclusion-consortium-design because the denominator defines the scope of the no-selective-prebiotic claim. ₁₂

Pathway and pair denominators. A pangenome table reports 16/16 conserved amino-acid pathways for 15 N. mucosa genomes while an earlier core profile reports 16/18, so the denominators differ. ₁₂ Relatedly, the competition rationale says the assay tested 3 commensal pairs against PA14 at multiple inoculation densities while the later table and conclusion refer to 5 unique pairs; both accounts describe pairs that were actually tested, so the discrepancy concerns how much of the pair space the test covered rather than whether pairwise testing occurred. ₁₂ The coverage still matters because formulation scoring assumes additive inhibition — the combination’s effect equals the mean of its members — so the number of pairs examined bounds how well synergy or antagonism among members was characterized, which bears on sample-size-aware-phenotype-consensus. ₁₂

Rounded variance shares. The discussion’s “approximately 27% of variance” for metabolic competition is shorthand for a four-feature multivariate metabolic model’s training R² = 0.274, not for the metabolic overlap–inhibition correlation alone (r = 0.384), and the methods section reports five-fold cross-validated R² = 0.145 ± 0.142 with identified overfitting in the 142-isolate cohort, so out-of-sample explanatory power is lower than the discussion implies. ₁₂ The discussion likewise rounds taxonomy’s additional explanatory variance to 9% where the methods section gives 8.6%, and bacteriocins, secreted enzymes and contact-dependent killing are proposed rather than established explanations for the strong positive residuals of the top formulation species. ₁₂

Outgrowth versus niche coverage. The discussion reports that a 3–5 organism consortium collectively covers 100% of PA14’s metabolic niche as an emergent community property, and asserts categorically that no individual commensal outgrows PA14 on any tested substrate. ₁₂ That categorical statement is in tension with the report’s own finding that only 13.8% of pairwise comparisons show a commensal rate advantage on PA14-preferred amino acid substrates; whether “outgrows” refers to a different metric or comparison set is not clarified, and niche coverage is in any case distinct from relative growth rate. ₁₂

Callable-compound count. The callable table lists salicylic acid, 3-hydroxybenzoic acid, 4-hydroxybenzaldehyde, phthalic acid and terephthalic acid as the Shikimates/Phenylpropanoids entries, while the interpretation states that six of the eight callable compounds route through well-characterized aromatic catabolism converging on protocatechuate/catechol and the β-ketoadipate pathway. ₇ The count discrepancy between table rows and narrative is unresolved. ₇

Dark-compound count. An interpretation passage in the same carbon census states that “The 75 dark compounds are not a failure of method”, while the report’s revised headline instead ranks the 74 dark compounds by why they are dark; the generated-data description identifies 75 as the pre-reclassification count, and both figures are retained as stated with no replacement computed. ₇

Confirmed-hypothesis counts. The functional-atlas report’s counts of confirmed pre-registered hypotheses disagree with one another: one passage reports “3 of 4 confirmed hypotheses survive D2 annotation-density residualization (P4-D5)” against a header count of 2 of 4 confirmed, the final synthesis reports 2 of 4 pre-registered weak-prior hypotheses confirmed while in the same passage referring to all 3 confirmed hypotheses, and the project synthesis states 3 of 3 confirmed hypotheses anchored in expected biomes and phenotypes, with pre-registered KOs described as not phage-borne and ICEs and chromosomal recombination implicated by literature. ₃ A further passage states that every confirmed pre-registered hypothesis (NB12, NB16, Phase 1B Bacteroidota PUL) is anchored to the expected biome at p < 10⁻¹¹, which conflicts with the final verdict table’s falsification of the Bacteroidota PUL hypothesis at its pre-registered criterion. ₃ All counts are retained as stated and none is reconciled; the falsified-at-criterion verdict is the one recorded above for PUL. ₃

Cocktail coverage targets versus achieved coverage. In the phage-targeting project’s set-cover table, the row labelled as a ≥95 % coverage target reports 94.7 % with five phages (DIJ07_P2 + LF73_P1 + AL505_Ev3 + 55989_P2 + LF110_P2) and the ≥99 % row reports 98.4 % with eight phages (adding NIC06_P2 + LF73_P4 + BCH953_P4), so neither target was met. ₂ The clinical hand-off nevertheless labels the five-phage cocktail as 95 % strain coverage and the eight-phage extension as 99 % strain coverage, comparable in size to EcoActive; those two figures must be read as hand-off labels or coverage targets, not as measured results. ₂ This bears on phage-therapy-evidence-translation, where the gap between a target and an achieved coverage is exactly what a downstream hand-off inherits. ₂

Phage inventory denominators. The same project’s synthesis row reports experimental susceptibility over 96 phages × 188 strains × 17,672 pairs, a five-phage cocktail covering 95 % of E. coli strains, and “69 % of phages AIEC-relevant phylogroup”. ₂ Its 96-phage total differs from the 94-phage denominator used elsewhere for B2/D phylogroup hosts, and its rounded 95 % differs from the achieved 94.7 %. ₂

Ecotype call inventory. HMP2 fact_viromics (3,039 sample-rows × 273 viruses × VirMAP taxonomic profile) cross-referenced with ecotype projections is reported as 630 of 648 viromics samples (97 %) with ecotype calls, listed as 27 (E0) + 484 (E1) + 55 (E2) + 50 (E3) samples spanning CD/UC/healthy; the listed per-ecotype counts do not match the stated 630 total, and no corrected total is computed here. ₂

Endogenous-phage correlation ceiling. The report finds no strong (|ρ| > 0.30) endogenous phage candidates targeting Tier-A pathobionts and states that all correlations are |ρ| ≤ 0.18, while its own table carries a value of +0.183, so the stated ceiling conflicts slightly with the tabulated value; the null result — no strong endogenous candidate — is unaffected by the discrepancy. ₂

Per-patient target totals. The per-patient table’s total row reports 23 patients, a mean of 4.39 targets per patient, 22 patients with cocktails and 14 with concrete phages, and the reported total mean does not align with the preceding per-ecotype means and counts; the discrepancy is recorded without recomputation. ₂

Replication of top candidates. The same report states that two top-20 candidates fail to replicate — S. thermophilus on effect sign, possibly cohort-specific dairy exposure, and B. stercoris non-significant — which conflicts with its adjacent claim that every top-ten candidate replicated; this bears on cross-cohort-microbiome-portability, where a single non-replicating sign flip changes which candidates transfer. ₂

Phage-coverable module members. The per-patient cocktail design reports that all 9 E1 patients carry the full 5-species Tier-A pathobiont module (H. hathewayi, F. plautii, E. bolteae, E. lenta, M. gnavus) while stating that only 3 of those 5 species have direct lytic phage options at present; the examples given — E. lenta PMBT5, E. bolteae PMBT24, E. coli AIEC if present — include a species outside that five, so the 3-of-5 count is internally inconsistent with its own examples. ₂ The hybrid-cocktail principle restates the count as E. coli, E. bolteae and E. lenta “when present” with H. hathewayi and F. plautii as the remaining Pillar-4 gap species, which leaves the stated module composition ambiguous; the structural conclusion it supports — a pure phage cocktail is infeasible for the dominant E1 ecotype, so the cocktail must be hybrid — does not depend on which species fills the third slot, and this bears on phage-therapy-evidence-translation. ₂

Phage-gap classification of M. gnavus. One passage lists H. hathewayi (universal 83 %) and M. gnavus (universal 91 %) as both Pillar-4 gap species representing the unmet need, whereas the project’s earlier availability table classifies M. gnavus as temperate-only and limited rather than a gap; both classifications are recorded as stated. ₂

Ecotype assignment of E. coli in cocktail strategies. The strategy table carries an “E1 hybrid 3-strategy (full, with E. coli AIEC)” row for 1 patient, which is in tension with the project’s proposed state-dependent rule treating E. coli inclusion as E3-specific. ₂

Synthesis figure versus its prose. NB17_synthesis.png describes its target decision matrix as 6 actionable Tier-A × 4 attributes while the accompanying NB17 prose gives 5 attributes; the same figure shows a UC Davis per-patient cocktail-design map of 4 design categories × 7 cocktail strategies and a 4-phase clinical-translation roadmap (Immediate / Near / Mid / Long term). ₂

Hypothesis verdict wording. The lab-versus-field ecology project’s novelty section calls H1 rejected, while its hypothesis-outcomes section calls the same hypothesis not supported but suggestive; both wordings are recorded, and this bears on lab-field-fitness-concordance, where “rejected” and “suggestive” license different follow-up. ₁₃

Figure-versus-prose percentages. A cross-project pitfall record notes a critical review issue in which figures showed 86%/78% while the prose said 82%/66%, attributed to an undocumented change of denominator. ₉ No figure number is supplied and no corrected pair is computed here; the recorded rule is to document which denominator is used and to stay consistent within a synthesis or comparison, which bears on pangenome-core-boundary-and-clade-size-bias. ₉

Susceptibility-matrix arithmetic. The defense-system project reports 188 E. coli strains (ECOR collection, NILS collection, clinical/environmental isolates) × 96 phages (41 Podoviridae, 32 Myoviridae, 23 Siphoviridae) = 17,672 binary infection outcomes, and that stated multiplication is internally inconsistent; the figures are retained as stated and no product is computed here. ₁₄ The same 96-phage × 188-strain × 17,672-pair inventory appears in the phage-targeting project, where a 94-phage denominator is used elsewhere, so the inconsistency is shared across reports rather than local to one. _2,14

Threshold-passing claim versus a tabulated odds ratio. The dark-genes project asserts that all effects exceed its pre-registered thresholds (Cohen’s d ≥ 0.2 or OR ≥ 1.5), while the preceding essentiality row gives OR = 1.420; both are retained as stated and the blanket threshold claim must not be carried forward as established. ₁₅

Convergence count. The plant-microbiome project reports that its original genus-level statsmodels logit “failed to converge for 0/14 markers”, a phrasing whose count conflicts with the stated failure, so the passage does not establish how many markers, if any, failed. ₈ The same passage treats the L1-bootstrap phylogenetic control as suggestive rather than confirmatory and closes the item with a cluster-robust generalized linear model (cluster = genus) offered as the pragmatic frequentist analogue of a phylogenetic generalized linear mixed model (PGLMM) at species scale; this bears on phylogenetic-confounding-of-pangenome-associations. ₈

Mechanism versus measurement

Adversarial review also separates what a project measured from what it interpreted. ₄ Pal upregulation at both transcript and protein level was measured, but an earlier draft’s explanation — that the Tol-Pal complex normally relies on lipopolysaccharide-mediated Mg²⁺-bridged outer-membrane–LPS–Pal stacking for outer-membrane/inner-membrane cohesion, so a cell with no LPS upregulates Pal to compensate by direct protein anchoring — had no cited support and was replaced. ₄ The replacement rests on Tan & Chng 2025 (Nature Communications 16(1):2293, PMID 40055349), which established that the primary function of the Tol-Pal complex is retrograde phospholipid transport to maintain outer-membrane lipid homeostasis rather than LPS-Pal structural anchoring. ₄

The replacement interpretation is an inference rather than a measurement: the data show Pal up, while the retrograde phospholipid-transport mechanism is grounded in the cited paper’s redefinition of Tol-Pal function and was not demonstrated by the project, and the adversarial review is what identified that citation as missing before it was added. ₄ This is the ordinary case for outer-membrane-lipid-homeostasis: a correctly cited mechanism remains a hypothesis until the project that invokes it tests it. ₄

The same gap appears where a transfer mechanism is inferred from the literature while its direct test is deferred. ₃ The functional atlas states that mycolic-acid, PSII (photosystem II) and PUL genes flow via chromosomal recombination and integrative and conjugative elements (ICEs) rather than via phage-mediated transduction, and that PSII gene-neighborhood composition at baseline rules out the alternative hypothesis that PSII gains travel as phage/MGE cargo. ₃ The PSII neighborhood result is a measurement, but the direct cargo tests for PUL and mycolic acid were deferred, so for those classes the mechanism statement is stronger than the project’s own direct evidence and rests on literature implication — the closing synthesis itself describes ICEs and chromosomal recombination as implicated by literature rather than demonstrated. ₃ This refines chromosomal-and-integrative-gene-transfer: for the untested cargo classes the mechanism is a hypothesis, which supports the hypothesis that the same non-phage route carries them. ₃

Data and provenance validation

Large KBase Data Lakehouse catalogs should be discovered through access-aware helpers such as get_databases(), get_tables(), and get_table_schema() rather than historical inventories. ₁ The live catalog included kescience_mgnify, PhageFoundry GenomeDepot databases, kescience_interpro, kescience_pubmed, pangenome_bakta, arkinlab_microbeatlas, protect_integration, and user-owned Klebsiella databases that were absent from older snapshots. ₁ This supports data-landscape-ownership-and-coverage-bias and cross-tenant-data-bridging: a reproducible claim must establish what data were available at analysis time, who owns the relevant resource, and whether the queried inventory was current. ₁

Local data marts benefit from lineage.yaml, schema_overview.yaml, per-table YAML dictionaries, and sentinel-code tables that distinguish pending data from true NULL values. ₁ These provenance artifacts make schema-to-value-space failures more detectable before they propagate into downstream biological conclusions. ₁ Large Spark joins touching the 2.5B-row UniProt identifier table should disable broadcast joins with SET spark.sql.autoBroadcastJoinThreshold = -1. ₁ Notebooks executed through nbconvert should cache large Spark outputs as CSV because direct toPandas() calls can cause DeadKernelError. ₁ These operational checks connect adversarial validation to cross-tenant-data-bridging: a theoretically available dataset is not equivalent to a successfully joined, interpretable, and reproducible evidence source. ₁

Validation protocol

A defensible claim should be challenged at four levels: citation identity, statistical specification, data provenance, and computational execution. ₁

1. Verify every load-bearing citation against author, title, journal, PMID, and DOI records in PubMed. ₁
2. Separate pre-registered hypotheses from post-hoc metrics and apply correction only to the relevant testing family. ₁
3. Re-run analyses with leakage checks, explicit confounder adjustment, empirical or simulated null distributions, and independently counted sample units. ₁
4. Record catalog discovery, table schemas, lineage, missing-value semantics, and join coverage before interpreting biological results. ₁
5. Test notebook-scale execution with cached intermediate outputs and join settings appropriate to table size. ₁

This protocol does not guarantee that a claim is true, but it makes unsupported citations, specification-dependent results, inaccessible data, and execution failures visible before they become biological conclusions. ₁

Relation to the wider corpus

Adversarial validation strengthens environmental-resistome by requiring marker definitions and database provenance to be explicit before resistance classifications are interpreted. ₁ It also complements metabolic-model-gapfilling and circularity-in-metabolic-model-validation by challenging whether model support, annotations, and validation data are independent. ₁ The same principle applies to cross-cohort-microbiome-portability, where leakage repair and independent cohort checks determine whether a classifier or biomarker transfers beyond its training data. ₁

Open Directions

• Use the 9 adversarial review rounds and their identified failure classes to build a blinded checklist study, then measure which checks most often overturn a claim. ₁
• Re-audit load-bearing citations against PubMed metadata and DOI records, then quantify the fraction of claims affected by PMID hijacking or hallucinated author lists. ₁
• Reconstruct the M1–M26 revision history with a hypothesis-family map, then test whether each correction belongs to a pre-registered family or a post-hoc metric family. ₁
• Execute representative KBase Data Lakehouse analyses from lineage files and schema dictionaries, then measure how often stale catalogs, sentinel codes, or incomplete joins change the result. ₁
• Benchmark cached CSV-based Spark workflows against direct toPandas() execution on joins involving the 2.5B-row UniProt identifier table, then identify reproducible thresholds for kernel failure and runtime degradation. ₁
• Re-test each pre-registered threshold against its own background rate — beginning with the ≥10% lipid-A threshold and the hypergeometric alternative adopted in its place — then record which pre-registered verdicts change. ₄
• Re-derive every KEGG Orthology marker set used for iron reduction by KO biological identity rather than by eggNOG-mapper similarity threshold, then report which prior subsurface comparisons retain significance after correction. _5,6
• Regenerate committed tables after each allowlist correction, starting with the R02107 exclusion, and recount callable compounds so that corrected rules and published numbers agree. ₇
• Audit narrative figures against their own source tables across projects — the single-predictor R² ordering, the 38.4% match rate, the 314-row universe file against the 334-genome cohort table, and the substrate and pair counts — then report how often a headline claim depends on the inconsistent number. _11,5,12
• Re-adjudicate the four pre-registered functional-atlas hypotheses under both the original absolute-zero criterion and M12’s relative criterion once its threshold X has actually been calibrated from Phase 2 data, then report which verdicts depend on which criterion. ₃
• Build the M20 independent paralog holdout from Pfam clans and KO orthogroups and re-validate the producer null outside the natural_expansion selection, then report whether the paralog-expansion signal survives a non-circular null. ₃
• Reconcile the functional atlas’s confirmed-hypothesis counts against its final verdict table, listing which pre-registered hypotheses are confirmed under the criteria as written. ₃
• Verify every reviewer-supplied citation against PubMed before acting on a review finding, then report the fraction of adversarial findings that rest on an unverifiable reference. ₃
• Recompute phage-cocktail coverage directly from the 188-strain susceptibility matrix and report achieved coverage rather than target labels, resolving the 96- versus 94-phage denominators at the same time. ₂
• Re-derive the per-ecotype viromics sample inventory and the per-patient target totals from fact_viromics, then state whether the subgroup counts match the reported 630-sample total. ₂
• Recount dark versus callable compounds after the reclassification and the allowlist fix, then publish one headline count per category. ₇
• Audit every permutation-based effect size in the corpus for the denominator error documented here — standard deviation of permutation means versus raw pair-level standard deviation — then report which reported effects shrink by more than an order of magnitude. ₈
• Re-run the complementarity test at reaction level or on substrate-specific sets rather than at the 80-pathway GapMind level, then report whether the small redundancy effect survives finer resolution. ₈
• Re-derive the E1 phage-coverage count from the strain-level availability table rather than from the per-patient narrative, then state which of the five Tier-A module members have direct lytic options. ₂
• Run the full three-method R-native ANCOM-BC / MaAsLin2 / LinDA consensus the plan specified, outside the ecotype-defined subsamples that bootstrap and LinDA share, and report whether the within-substudy candidate set changes. ₂
• Re-test the NB08a iron-siderophore enrichment against a matched-niche pathobiont comparator at similar genome-assembly depth, then report the odds ratio under that background rather than against the full catalog. ₂
• Add dtype and schema assertions for the recorded silent-failure patterns — object-dtype boolean flags, suffix-only p-column selection, copied feature frames, reused test variables and non-existent score categories — then measure how many committed analyses change their numbers. ₉
• Extend the synthetic-data ordering test to the other phylogeny-aware models in the corpus, then report which fitted results depend on input row order. ₁₀
• Populate phylogenetic-tree distances for the 47 uncovered candidate species and re-run the within-species subclade scan, then report the H7 effect across the full 65-species candidate set. ₈