Selection-on-outcome leakage occurs when observations are grouped or selected using the same biological features that are later tested for differences, so the analysis can amplify apparent associations and overstate their reproducibility. 1 The documented microbiome example shows that clustering taxa and then testing those taxa within the resulting clusters can produce unstable candidate lists. 1 This concept connects pitfalls and Metagenome-Prioritized Phage Cocktails for Crohn’s Disease and IBD with ecotype-environment-gene-content, confirmatory-exploratory-ecological-association-discordance, and sample-size-aware-phenotype-consensus.
Mechanism
In the cited workflow, samples were clustered on a taxon-abundance matrix, and the same taxa were subsequently tested within the clusters. 1 The general pattern is to define ecotypes or clusters by unsupervised partitioning of the taxon abundance matrix and then to run differential abundance (DA — testing whether a feature’s abundance differs between groups) within each cluster. 1 Because the taxa influence both cluster membership and the test outcome, the apparent within-cluster differences are not independent evidence for the clusters or for the selected taxa. 1 The consequence is selection-on-outcome confounding: within-cluster effect sizes are mechanically inflated for cluster-defining taxa, and the within-cluster top-N list is substantially the cluster definition itself. 1,2 This is a form of outcome-dependent feature selection: the analysis uses a feature to define the comparison and then treats the feature’s contrast as if it had been selected independently. 1
The problem is not limited to microbiome taxa; the same design risk applies whenever genomic or functional features are used to construct groups and are then reused as outcomes. 1 The Ecotype Functional Differentiation supports this genomic extension: PCA and KMeans clustering of within-species gene content was followed by testing COG (Clusters of Orthologous Groups) functional profiles, which summarize overlapping gene-content variation. Across 12 species, 170 of 257 COG tests (66.1%) were significant after BH-FDR correction, but this demonstrates functional differentiation rather than independent confirmation of it because the tested representation is biologically related to the grouping representation. 3 Independent validation, held-out features, or a genuinely different data representation is therefore needed before treating selected associations as confirmatory findings. 1
The same bug appears wherever cluster labels are a function of the features being tested, including microbiome enterotypes and single-cell type-stratified differential expression, where clustering on gene expression and then testing gene-level differential expression within cluster is the single-cell analog. 1
Evidence from the IBD ecotype analysis
In ibd_phage_targeting, K=4 latent Dirichlet allocation (LDA) ecotypes were trained on MetaPhlAn3 species abundances and within-ecotype CD-vs-HC CLR (centered log-ratio) Mann-Whitney tests were then run on the same species matrix, producing a 33-species Tier-A list with CLR-Δ effect sizes from +0.5 to +3.0. 1 That list comprised 18 E1 and 15 E3 candidates, and the H2c C. scindens paradox was marked “RESOLVED by stratification” before any repair filters were applied. 4
Leakage was measured two ways. Held-out-species sensitivity used 5 random 50/50 species splits, refitting the LDA ecotypes on half A, re-testing half B within the refit ecotypes, and computing the Jaccard overlap between the refit top-30 and the original top-30 intersected with half B; the stated bounds were > 0.5 = leakage bounded and < 0.3 = leakage dominates. 1,2 Observed values were E1 = 0.230 and E3 = 0.064, both below the < 0.3 bound, which the project read as confirming that the within-ecotype Tier-A list was substantially leakage-driven. 1,2 Leave-one-species-out refitting, the second diagnostic, refit the ecotypes on all other species per battery species, re-derived the cluster labels, and re-tested the held-out species. 1
Leave-one-species-out refitting changed C. scindens from a non-significant result to CD increases in both E1, with CI +0.68 to +0.87, and E3, with CI +1.13 to +1.71. 1 The revised report retracts the original H2c claim that within-ecotype stratification resolved a C. scindens paradox: under the confound-free within-IBD-substudy CD-vs-nonIBD meta-analysis the species is genuinely CD↑ with pooled CLR-Δ = +1.18, FDR (false discovery rate) = 1e-8, and 4/4 sign concordance across sub-studies, and under leave-one-species-out refit it is CD↑ in both E1 and E3 once it is not part of the clustering input. 2,4 This contradicts the earlier within-ecotype “n.s.” call, which the report attributes to feature leakage rather than to a real ecological paradox. 2 The reversal supports the interpretation that reusing clustering-related taxa can alter both significance and estimated direction, rather than merely reducing statistical power. 1
The 33-species Tier-A list is likewise retracted. 2 The held-out-species sensitivity values (E1 = 0.230, E3 = 0.064, against the > 0.5 leakage-bounded threshold) together with the finding that 14 of 18 E1 candidates had negative within-substudy effects under an independent confound-free CD-vs-control contrast identified those candidates as ecotype-markers rather than CD drivers. 2 The original within-ecotype CD-vs-HC CLR differential-abundance claims 1 and 2 are marked superseded and kept only for audit. 2
Two rigor-repair notebooks applied three evidence filters to every candidate; the third required CD↑ with at least 66 % sign concordance in a within-substudy CD-vs-nonIBD meta-analysis across 4 IBD sub-studies, used as a confound-free independent check. 4 Only that within-substudy stream is independent of the ecotype definition that generated the original list. 4 Under the filters, 0 of 18 E1 candidates passed all three, and all 14 E1 candidates that passed the first two had negative effects under the confound-free contrast — they are ecotype-markers, not CD drivers, which the digest describes as the within-ecotype DA having “lifted” them to CD↑ through a compound of feature leakage and compositional bias. 4 An independent-design cross-check reported in pitfalls reaches the same conclusion: most of the Tier-A E1 candidates had negative within-study effects and are ecotype-markers rather than CD drivers. 1
Species that passed both the possibly leakage-contaminated within-ecotype DA and an independent CD-vs-control contrast were treated as the trustworthy Tier-A. 1 Independent within-substudy evidence reduced the Tier-A list from 33 candidates to 3. 1 This sharp reduction supports the use of independent validation when exploratory cluster-derived associations are being considered for confirmation. 1
The confound-free analysis flatly contradicts the within-ecotype calls for several species: F. prausnitzii, R. hominis, and L. eligens were called CD↑ within both E1 and E3 in the original “Simpson’s paradox” section but are CD↓ under the confound-free contrast, consistent with their classical protective-commensal role, and the C. scindens “n.s.” call is similarly contradicted. 2 The report attributes these direction reversals to a compound artifact of feature leakage plus the pooled-cohort substudy × diagnosis confound, and notes that both compositional-bias-aware DA methods tried on the within-ecotype subsets (CLR-Mann-Whitney and LinDA) share the bias, so a count of evidence drawn from within-ecotype methods alone does not resolve it. 2
Bootstrap stability does not fix the problem. Bootstrap-stable within-ecotype DA and LinDA operate on the same ecotype-defined subsamples and therefore share the selection-on-outcome bias of the original CLR-Mann-Whitney analysis, leaving the within-substudy CD-vs-nonIBD contrast as the only independent evidence source — which is why the Tier-A gating requires within-substudy concordance rather than counting bootstrap and LinDA as separate evidence streams. 2
Detection by adversarial review
Leakage in this corpus was found by adversarial review rather than by routine checking, linking this concept to adversarial-research-quality-assurance. A general-purpose agent instructed to find flaws on the same files identified 5 critical issues plus 6 important issues, including feature leakage in within-cluster DA where the analysis clustered samples on taxon abundances and then tested the same taxa within cluster; hard-coded “RESOLVED” verdict logic that treated n.s. results as positive evidence; zero confounder adjustment despite citing a reference that calls for it; a Jaccard 0.14 reported as supporting a hypothesis without any null distribution; and an ecotype framework with 48.9 % cross-method agreement and no external replication. 4
The Jaccard criticism was then closed rather than left open, which refines the review finding above: a permutation null for Jaccard(top-30 E1, top-30 E3) under randomized ecotype labels yields mean 0.785 ± 0.054, and the observed Jaccard = 0.104 gives empirical p = 0.000 over 200 permutations, so the E1 and E3 target sets genuinely diverge rather than overlapping at random. 2 The report retracts the original “Jaccard = 0.14 supports H2b” framing because that value was near the random-overlap baseline and had no null distribution attached, while stating that the H2b conclusion survives even though the statistic did not. 2
A later adversarial review of plan v1.6 found 4 critical and 10 important issues structurally analogous to the earlier failure pattern: feature leakage in an N14 dual-basis analysis whose same-axis ecotype basis is leakage-poisoned; a “4-substudy meta-viable” claim inherited without re-verification for pathway data (actually 3+1); thresholds without null distributions; a mislabeled “butyrate-producer ↔ pathobiont” hypothesis with 4 of 6 named anchors not actually butyrate producers; a strain-adaptation hypothesis that collapses to species level on curated metagenomic data by construction; and an n=67 across 3 sites presented as a 130-subject HMP2 analysis. 4
Same-axis feature leakage recurred on both occasions: clustering on taxa and testing taxa in the earlier notebook, and stratifying pathway differential abundance by a pathway-feature ecotype in the later plan. 4
Designs that remove the leakage
Leakage should be assessed with held-out-feature clustering, leave-one-feature-out refitting, or clustering on a genuinely different functional matrix such as pathways or EC numbers. 1 Defining clusters on a functional matrix (KEGG pathways, MetaCyc, EC numbers) and then testing taxonomic DA makes the two matrices genuinely different, so leakage becomes structurally impossible rather than empirically bounded. 1 Rebuilding the ecotype analysis on a functional feature matrix was flagged as future work rather than carried out. 1 These designs separate the features used to define the ecological comparison from the features used to evaluate it, reducing direct reuse of the selected outcome. 1
The replacement design stratifies the within-substudy CD-vs-nonIBD contrast by ecotype before meta-analysis, testing whether the canonical CD signature differs by ecotype while producing ecotype-specific Tier-A lists that are structurally free of leakage; three (substudy × ecotype) cells met the eligibility bar of ≥ 10 CD and ≥ 10 nonIBD samples. 2 In this form the clustering axis and the DA axis are disjoint — samples are partitioned by ecotype and CLR-Δ is computed across sub-studies within a partition — which the report calls structurally immune to the leakage, even while documenting that the original within-ecotype Tier-A cannot be trusted directly because the first criterion fails in both ecotypes. 2
The repair pipeline that produced the replacement analysis added bootstrap confidence intervals, leakage-bound sensitivity, leave-one-out refits, a Jaccard permutation null, and ecotype-stability checks, followed by a confound-free within-IBD-substudy CD-vs-nonIBD meta-analysis with LinDA implemented in pure Python, a rigor-controlled stopping rule, and a within-ecotype × within-substudy meta-analysis; it recorded substudy-nesting unidentifiability and feature leakage in cluster-stratified DA as two generalizable pitfalls. 2 The same repair documented linear mixed-effects (LME) unidentifiability in the pooled design. 2
The leakage-free design yields usable output rather than only a negative verdict: the within-IBD-substudy CD-vs-nonIBD meta-analysis resolves both the feature-leakage and substudy-confounding pitfalls of pooled-cohort DA and produces six actionable Tier-A targets — H. hathewayi (4.0), M. gnavus (3.8), E. coli (3.6), E. lenta (3.3), F. plautii (3.3), E. bolteae (2.8). 2 The project lists feature leakage in cluster-stratified DA alongside cMD substudy-nesting unidentifiability, leave-one-substudy-out adjusted Rand index (ARI) as an honest stability metric, pool ≠ flux for pathway versus metabolite DA, multi-line cross-corroboration as a portable rigor pattern, and cohort-batch-dominates-clustering for absolute-intensity multi-cohort modalities as generalizable methodology contributions. 2 The scope of the invalidation is bounded: the ecotype framework itself remains usable for patient stratification, and only the within-ecotype differential abundance was invalidated for target-selection use, which refines rather than overturns gut-microbiome-ecotypes-as-patient-strata. 4
Removing the leakage does not make the replacement lists confirmed. The ecotype-specific E3 Tier-A of 40 candidates comes from the single study HallAB_2017 (22 CD / 31 nonIBD) and is flagged as provisional, needing replication, and E3-specific replication requires a second cMD-IBD study that populates E3 with both CD and nonIBD. 2 The report further states that Pillar 2 cross-cohort replication is weak, and that single-study E3 evidence combined with marginal ecotype stability means the Tier-A list — particularly E3 — should be treated as hypotheses for further experimental validation rather than established targets. 2
Interpretation and adjacent null evidence
The findings should be interpreted as evidence about analysis design, not as evidence that all ecotype-associated taxa are false discoveries. 1 The document instead shows that the original 33-species list was sensitive to feature reuse and that independent within-substudy analysis provided a more restrictive set of 3 candidates. 1 This refines confirmatory-exploratory-ecological-association-discordance by identifying feature reuse as a concrete source of discordance between exploratory and confirmatory ecological associations. 1
Leakage safeguards also matter for ecotype-environment-gene-content, because ecotype-linked environmental or genomic interpretations can inherit instability when the grouping variables are tested again as explanatory features. 1 The Ecotype Reanalysis — Environmental vs Human-Associated Species refines this caution: using genome-level environmental classifications and a consistent within-method comparison, environmental species had no stronger environment–gene-content correlations than human-associated species (one-sided Mann-Whitney U: U=1536, p=0.83), and the clinical sampling bias did not explain the weak signal. 5 This distinguishes selection or composition effects from outcome leakage rather than establishing that either is absent. 5 The reanalysis also used a different genome-extraction and downsampling strategy, so its absolute correlations are not comparable with the original analysis; only the within-method group comparison was treated as valid. 5
The independent ecotype_analysis supports separating leakage from a genuinely weak environmental signal: across 172 bacterial species, the median partial correlation for environment was 0.0025 versus 0.0143 for phylogeny, with no significant environmental effect in 156 species (90.7%). 6 It therefore suggests that weak whole-genome environmental associations cannot, by themselves, demonstrate outcome leakage; the study instead hypothesized that environmental adaptation may affect specific gene subsets and that AlphaEarth embeddings may miss relevant variation. 6 AlphaEarth embeddings covered only 28.4% of genomes, making limited environmental coverage an alternative explanation that should be distinguished from feature reuse. 6
Those independent genome-level comparisons complement, rather than replace, design-consistent within-substudy contrasts, because the documented healthy and disease buckets contained disjoint sub-studies and pooled mixed-model contrasts were structurally unidentifiable in that setting. 1
Related forms of circularity
The same design logic applies when a predicted score stands in for a measured phenotype: Metal Fitness Atlas scores are genome-based predictions, not direct metal tolerance measurements, and controlling for n_metal_clusters in partial correlations mitigates circular reasoning but leaves the phenotype-score associations as phenotype-to-genome correlations rather than phenotype-to-phenotype ones. 7 This caveat is shared with composite-resistance-score-limitations. 7
A classification threshold can also make a grouping partly definitional. Of the 52 ICA (independent component analysis) fitness modules, the mean core fraction is 0.886 and the median is 1.000, and the module types were defined using the mean core fraction, 0.886, as the classification threshold rather than the median; the low conservation of the nine “lab” modules (mean core fraction 0.516) is therefore partly a consequence of the classification criterion rather than an independent result. 8
Shared composition produces the same inferential hazard without any clustering step. In the NMDC carrier–trait analysis, the extremely high significance likely reflects compositional coupling, because genera abundant in a sample contribute to both the carrier abundance score and the community trait score. 9 In a plant-microbiome marker analysis, a random forest using 25 binary markers predicted compartment (root, rhizosphere, or phyllosphere) with 64.4% accuracy, above the 33% random baseline but far from deterministic, whereas cohort prediction reached 99.9% trivially because cohorts are defined by marker presence — a classifier scoring its own definition. 10
Tensions
The evidence does not establish a universal Jaccard cutoff for detecting leakage. 1 The values 0.5 and 0.3 were explicitly project-specific decision thresholds for the cited sensitivity procedure, so applying them unchanged to another dataset or clustering method would be an unsupported extrapolation. 1
The observed instability also does not by itself identify whether clustering, differential-abundance modeling, subgroup sample size, or study structure contributed most to the changes. 1 Resolving those components requires analyses that vary the feature partition and validation design while preserving the underlying samples and labels. 1
The ecotype functional-differentiation result refines this tension rather than resolving it: all 12 analyzed species had at least one differentiated COG category, but approximately 38% of gene clusters had COG annotations, and the report notes that effect significance may partly reflect large sample sizes. 3 Thus, widespread COG differences are compatible with real functional structure, annotation and sampling effects, or reuse of related gene-content information; they do not alone quantify leakage.
The ecotype reanalysis further shows that absolute partial-correlation values can differ substantially when genome inclusion and downsampling change: its median across 183 species was 0.081 versus 0.003 in the original analysis, described as a 27x difference. 5 This supports retaining the existing warning against interpreting instability or magnitude across incompatible methodologies as evidence of leakage alone, while the within-method environmental-versus-human-associated null comparison remains informative. 5
The original ecotype analysis reports a weak environmental signal overall, whereas the reanalysis reports a median partial correlation of 0.081 under a different pipeline. 6,5 This supports treating cross-pipeline magnitude comparisons as a tension in measurement and sampling rather than as evidence that leakage caused either result. 6,5
Open Directions
- Re-run the IBD ecotype analysis with held-out-species clustering and compare Tier-A membership, CLR-Δ estimates, and confidence intervals with the original 33-species list; determine which associations persist without direct feature reuse. 1
- Report the per-species leave-one-species-out refit outcomes for the whole battery rather than citing the C. scindens reversal alone; quantify how often significance, effect direction, and ecotype-specific estimates change. 1
- Cluster the same samples using pathways or EC numbers instead of taxa, then test taxon-level associations; determine whether a feature representation that is independent of the tested taxa reduces the leakage signal. 1
- Combine the four IBD sub-studies with at least 10 CD and 10 nonIBD samples using within-substudy contrasts and inverse-variance meta-analysis; compare the resulting candidates with the ecotype-derived list. 1
- Recalculate stability across the reported Jaccard values of 0.230 for E1 and 0.064 for E3 under alternative feature-holdout partitions; determine whether the observed instability is specific to the cited partitioning procedure. 1
- Extend the within-ecotype × within-substudy design beyond the three eligible (substudy × ecotype) cells and replicate the provisional single-study E3 Tier-A in a second cMD-IBD study that populates E3 with both CD and nonIBD; determine whether any ecotype-specific target survives a design in which the clustering and DA axes are disjoint. 2
- Reconcile the 27x partial-correlation discrepancy by extracting downsampled and full-genome sets under the same pipeline, then test whether genome count and feature reuse jointly alter environmental-versus-human-associated comparisons. 5
- Test whether COG functional categories and alternative environmental metadata reveal associations masked at whole-genome resolution without reusing features to define groups. 6
- Reanalyze ecotype clusters with held-out COG categories and within-species phylogenetic controls; test whether the 170/257 significant COG contrasts persist independently of gene-content clustering and phylogenetic structure. 3
- Replicate the ecotype framework in an external cohort to close the remaining gap named by adversarial review — 48.9 % cross-method agreement with no external replication — noting that the Jaccard gap is already closed by the permutation null (observed 0.104, empirical p = 0.000 over 200 permutations). 4,2
- Re-test the Metal Fitness Atlas phenotype associations against directly measured metal tolerance instead of genome-based scores; determine how much of the association survives when the predictor is not derived from the same genome. 7
- Reclassify fitness modules using the median core fraction, 1.000, instead of the mean, 0.886, and test whether the low-conservation “lab” class of nine modules remains distinct under the alternative threshold. 8
- Recompute the NMDC carrier–trait correlations with carrier genera excluded from the community trait score; quantify how much of the reported significance is compositional coupling. 9
- Validate the 25 binary compartment markers on cohorts that are not defined by marker presence; determine whether the 64.4% compartment accuracy holds once the cohort labels are independent of the markers. 10