Cross-cohort portability is the ability of a microbiome classifier or metabolomics-based analysis to preserve its biological interpretation when applied to samples collected in a different study, laboratory, platform, or feature namespace. Evidence from discoveries shows that portability depends strongly on representation: relative-abundance features transferred more robustly than absolute-intensity metabolomics, while cohort effects could overwhelm disease or ecological signals. 1
Evidence
Pooling hmp2 and franzosa metabolomics on 122 m/z-bridge metabolites produced principal-component analysis (PCA) plus K-means clustering with K=4 that separated completely by cohort rather than diagnosis. 1 The first principal component explained 79% of variance, and leave-one-study-out (LOSO) adjusted Rand index (ARI) was 0.000 for the metabolomics representation versus 0.113 for taxonomic ecotypes. 1 These results support treating absolute metabolite intensities as cohort-sensitive measurements rather than directly portable biological features. 1
Relative-abundance spaces were more cross-cohort-portable because they are unitless and compositional, whereas absolute-intensity metabolomics required explicit batch correction such as ComBat, surrogate variable analysis (SVA), RUV, or quantile normalization. 1 This refines cross-cohort-microbiome-portability from a classifier-performance problem into a measurement-scale and preprocessing problem. 1 The Web of Microbes Data Explorer provides adjacent evidence that portability also depends on cross-collection feature identity: among 257 identified Web of Microbes compounds, 69 (26.8%) had definitive ModelSEED links, while formula-only matching yielded 107 compounds expanding to 900 candidate molecules. 2 This supports treating namespace harmonization and match confidence as portability requirements, not merely implementation details. 2
The bridge supporting the metabolomics comparison was built within method: HMP2 named metabolites, which carry HMDB identifiers with m/z and retention time, were matched to franzosa peaks by m/z at ±0.005 Da tolerance, and retention time was deliberately excluded because the two runs used different gradients and columns; Franzosa metabolite IDs follow a [m/z]_[RT] format with no compound-name annotations in the mart. 3 Of 592 named HMP2 metabolites, 122 matched a Franzosa peak and 118 of those 122 passed Franzosa CD-versus-Control differential-abundance filters. 3 The comparison cohort, FRANZOSA_2019, is an independent multi-site IBD metabolomics cohort of 220 participants × 88 CD + 76 UC + 56 Control, while a candidate mouse ileal AKR/J tissue metabolomics dataset (DAVE_SAMP_METABOLOMICS) was dropped as not human IBD. 3
That cohort-driven clustering was the answer to an explicit hypothesis test, not an incidental observation: the project asked whether metabolite-derived clustering achieves higher cross-cohort stability than the taxonomic ecotype framework, since if it did, refitting ecotypes on metabolite features would give a more cross-cohort-portable basis for clinical translation. 3 Pooled HMP2 + Franzosa metabolite-feature K=4 K-means on the m/z-bridge panel (326 subjects × 111 features) instead produced structure that was cohort-driven rather than diagnosis-driven, with PC1 explaining 79 % of variance and separating HMP2 from Franzosa cleanly; leave-one-study-out ARI was 0.000 with either cohort held out, against the 0.113 taxonomic baseline, while the within-pooled bootstrap (80 % × 30) ARI of 0.937 versus the taxonomic 0.16 was called misleading because it measures cohort-batch reproducibility, not biology. 3 The tested metabolite-feature framework therefore did not achieve higher cross-cohort stability than the taxonomic framework, which remained that project’s primary ecotype basis. 3
Three scope conditions bound that negative verdict and keep it from generalizing into a claim that metabolite features are inherently non-portable: the m/z-bridge feature space holds only 111 unique features, and a larger space of 500-1000 metabolites with name annotations across cohorts might give different results — but obtaining it requires external Franzosa supplementary annotations that the project’s plan placed out of scope; no batch correction was applied, so the verdict stands for raw m/z-bridge clustering and ComBat, SVA or RUV would likely improve cross-cohort LOSO ARI substantially; and K=4 was plan-imposed to match the taxonomic framework when the natural cluster count on this feature space might be K=2, HMP2 versus Franzosa. 3 This refines rather than contradicts the digest’s reading, which likewise warns that within-pooled bootstrap stability metrics measure batch reproducibility rather than biological reproducibility when batch dominates, and that a within-cohort metabolite-feature ecotype framework may still be useful for clinical translation inside a single laboratory’s analytical pipeline — the failure is specifically the cross-cohort-portability question. 1
Metabolite-level replication, as distinct from clustering, partly succeeded on the same bridge. Across the 118 testable matches, overall sign concordance was 64 % (76 of 118), moderately above the 50 % chance baseline, and 9 matches counted as strict cross-cohort replications, meaning each passed false discovery rate (FDR, the expected share of false positives among called hits) < 0.10 and |cliff| > 0.20 in both cohorts and agreed in sign; the reported verdict was strong cross-cohort replication on 3 of 8 testable themes. 3 The theme table reports matched count, CD-up count in HMP2, CD-up count in Franzosa, sign-concordant count, percent concordance and median Franzosa Cliff’s delta (a rank-based effect size): urobilin_porphyrin 3, 0, 0, 3, 100 %, −0.747 (CD-DOWN); acyl_carnitines 5, 4, 5, 4, 80 %, +0.531; long_chain_PUFA 12, 12, 9, 9, 75 %, +0.273; aromatic_AA_metabolites 3, 3, 2, 2, 67 %, +0.181; bile_acids_primary 7, 5, 4, 4, 57 %, +0.252; lipid_classes 21, 14, 4, 11, 52 %, −0.069; tryptophan_indole 2, 1, 0, 1, 50 %, −0.199; bile_acids_secondary 5, 3, 2, 2, 40 %, −0.159; polyamines 0 matched, with no m/z match recorded as a methodological gap. 3 The three signatures at or above 75 % concordance — urobilin CD-down, acyl-carnitines CD-up, and long-chain polyunsaturated fatty acids (PUFAs) CD-up — are the cross-cohort-replicated metabolite set the project reports. 3
Each of the three carries its own effect size and its own interpretive distance. Urobilin was CD-down in 3/3 matches at median Franzosa cliff −0.747, the largest cross-cohort effect in the panel, read as loss of bilirubin-reducing commensals. 3 CD-up acyl-carnitines replicated in 4/5 matches at median cliff +0.531, including the C16 and C18:1 long-chain species, with impaired β-oxidation in inflamed mucosa offered as mechanism rather than measured. 3 Long-chain PUFAs replicated in 9/12 matches at +0.273, with adrenate, arachidonate, DHA and DPA replicating cleanly while eicosapentaenoate showed the opposite sign in 1 of 3 non-concordant cases — cohort-specific dietary fish-oil supplementation raised only as a possibility. 3
Bile-acid signals were weaker across cohorts than within them: primary bile acids replicated at 57 % (4/7) and secondary bile acids at 40 % (2/5), the latter below chance. 3 This refines rather than removes the bile-acid network claim by ranking evidence streams — paired within-HMP2 species × metabolite correlations, ρ(flavonifractor plautii × cholate) = −0.26 and × lithocholate = +0.15, are treated as the more robust stream, since bile-acid pool sizes are more susceptible to dietary and antibiotic confounding than within-cohort species × metabolite correlation, and cohort-aggregate cross-cohort metabolite differential abundance is dominated by batch effects. 3
One theme could not be tested at all. m/z-only matching recovered none of the 9 polyamines that drove the HMP2 theme enrichment at OR = 14.6, because these are low-mass hydrophilic-interaction-chromatography positive-mode compounds (putrescine 89.11; spermidine 146.17; spermine 203.22) and the m/z tolerance plus Franzosa method-library overlap yields no matches. 3 This is a feature-coverage gap rather than a failed biological replication: the polyamine OR = 14.6 stands as a single-cohort finding until external Franzosa supplementary tables with compound-name annotations become available, which lies outside the project’s no-raw-reads scope. 3 The project’s own summary of the bridge pairs this gap with the secondary-bile-acid result: the polyamine theme cannot bridge by m/z because of the low-mass hydrophilic-interaction-chromatography-positive gap, and secondary bile acids reach only 40 %, where cohort-aggregate differential abundance is the weaker evidence stream relative to the paired sample-level analysis. 3
The replicated PUFA signal also surfaced in a multi-omic projection, loading positively on the first canonical correlate (CC1): docosapentaenoate twice (+0.092, +0.091), adrenate +0.081 and arachidonate +0.048, reported against the long-chain-PUFA OR = 7.9 and the 75 % cross-cohort concordance. 3
Plant-associated comparative analyses support this representation warning at ecological scale. Pangenome-derived plant genera and MGnify rhizosphere genera had only 11.7% Jaccard overlap because isolation metadata and metagenomic detection measure different phenomena; the report therefore shows that apparently similar cohort labels can define materially different feature spaces. 4 Plant compartment effects also became much smaller after controlling for taxonomic composition: the refined 17-marker analysis gave PERMANOVA R² = 0.071, while db-RDA location-only R² = 0.060, and the earlier R² = 0.527 fell to R² = 0.072 after excluding three genome-rich species per compartment. 4 This supports the existing conclusion that pooled ecological separation can reflect dominant taxa or sampling composition rather than a portable compartment classifier.
A K=4 consensus of Latent Dirichlet Allocation (LDA, a topic model fit here on pseudo-counts) and a Gaussian mixture model (LDA-GMM) applied to 8,489 curatedMetagenomicData samples identified E0 diverse commensal (n=3,604), E1 Bacteroides2 transitional (n=2,601), E2 Prevotella copri enterotype (n=920), and E3 severe Bacteroides-expanded (n=1,364). 1 Healthy samples concentrated in E0 plus E2 at 84% combined, whereas Crohn’s disease and ulcerative colitis samples distributed across E1 and E3. 1 Cross-method validation selected K=4 because ARI was 0.131, within 0.02 of the peak at K=7 with ARI=0.140. 1
Portability failures remained after apparently strong pooled validation. A classifier trained on pooled HC + IBD samples to predict K=4 ecotype from {is_ibd, sex, age} achieved macro one-versus-rest area under the ROC curve (AUC) of 0.80 in 5-fold cross-validation, exceeding its 0.70 “passes” threshold, but reached only 41% agreement on UC Davis patients. 1 In that external cohort, is_ibd=1 was constant and 19/22 patients were predicted as E1, the marginal mode among IBD training samples, because the dominant learned rule was “IBD → E1 most of the time”. 1 The two tasks are not the same problem: the AUC counted HC-versus-IBD separability toward its score, whereas the UC Davis test requires the harder IBD-internal separation of E1 transitional from E3 severe. 1 This contradicts the assumption that pooled AUC alone demonstrates transferability and supports external-cohort evaluation as a required validation step — concretely, applying the classifier to an independent cohort with the cohort-axis variable held fixed and quoting the resulting agreement rate as the translation-honest metric. 1 The project report supports the same conclusion operationally: clinical covariates alone cannot assign IBD patients to ecotype, the classifier collapses to “IBD → E1” once is_ibd becomes constant on the test cohort, and metagenomics remains required for patient-level ecotype assignment in an all-CD cohort. 3
The project’s own figures are finer-grained than the rounded digest number. The minimal classifier on {is_ibd, sex, age} at n = 8,489 reached macro one-versus-rest AUC = 0.799, and an extended classifier adding {hbi_max, sccai_max, calp_max} on an n = 1,675 subset reached macro AUC = 0.810. 3 Against the metagenomic ecotype calls on UC Davis, the minimal classifier agreed on 41 % (9/22) of patients and the extended classifier on 36 % (8/22), with 12 / 22 patients disagreeing under both. 3
The mechanism is visible in the training marginals rather than in the test cohort. The minimal classifier predicted E1 for 19/22 UC Davis patients because training IBD samples split ~58 % E1 / ~40 % E3 / ~2 % E0 / ~0 % E2, making “is_ibd = 1 → E1” the dominant learned rule, which collapses to the marginal mode when is_ibd is held constant; the extended classifier’s training subset of 702 E1 / 959 E3 / 3 E0 / 11 E2 is effectively an E1-versus-E3 binary that severity markers do not separate reliably. 3 The report generalizes this into a reusable diagnostic: the gap between one-versus-rest AUC and per-patient agreement flags classifier-utility overstatement whenever a cohort-axis variable dominates the features, and it names its own H1c result — is_ibd dominating classifier AUC and collapsing on a single-cohort test — as a concrete instance of that problem. 3
External replication of the framework itself used HMP_2019_ibdmdb, pulled via pull_hmp2_metaphlan3.R against curatedmetagenomicdata v3.18 and deliberately not part of the cMD-IBD training set; HMP_2019_ibdmdb was absent from the CMD_IBD substudy list, so it is a genuinely held-out cohort that nonetheless shares the same metaphlan3 classifier namespace as the training data. 3 The pull returned 1,627 samples, 130 subjects and 582 species, with 255/335 training-feature overlap after synonymy. 3 Projecting 1,627 samples from 130 subjects onto the K=4 LDA placed 80.4 % of samples at confidence > 0.70 — median maximum LDA posterior 0.861 — and the subject-level ecotype × {CD, UC, nonIBD} association was χ² = 15.61, p = 0.016, so ecotype stratified disease in that independent cohort at significance; the formal external-replication verdict recorded for this analysis is PASS. 3 Subject-level assignments concentrated in E1 at 106 subjects (82 %), with E2 11, E3 10 and E0 3; the report’s reading that this E1 skew reflects HMP2’s recruitment of newly-diagnosed or milder IBD rather than flare-dominated severe disease is offered as a likely explanation, not a tested one. 3 The report’s own summary of these analyses holds both halves together: its stability tests show real cross-study and cross-feature-basis ecotype variance, while the same 1,627-sample projection onto the K=4 LDA recovers non-random disease stratification (χ² p=0.016) at high projection confidence (80 % of samples with maximum posterior > 0.70) and the operational E1 Tier-A list replicates at 88.2 % sign concordance on HMP2 — so the framework is described as usable even though its ecotype boundaries are not bit-reproducible across studies. 3 Across the project the consensus K=4 framework was applied to 8,489 cMD samples, 1,627 HMP2 samples and 23 UC Davis patients, with 80% HMP2 projection confidence and χ²(2)=15.6, p=0.016 for disease stratification. 3 The framework nonetheless carries real cross-study variance — leave-one-study-out ARI 0.113 — even while its operational Tier-A candidate list is externally validated; this refines the portability framework by showing that high projection confidence and weak cross-study cluster stability are compatible and must both be reported. 3
That 0.113 deserves full resolution. Across the top 8 curatedMetagenomicData sub-studies, mean leave-one-substudy-out ARI was 0.113 with range 0.000–0.282 and mean per-sample agreement 55.0 % with range 16.9 %–85.5 %, failing the project’s > 0.30 “stable” threshold; the formal LOSO verdict recorded is “study-dependent”. 3 This contradicts the more optimistic within-pooled bootstrap ARI of 0.13–0.17, which the report states understated the instability. 3 Alignment is also not uniform across receiving studies: LifeLinesDeep reached ARI 0.21 with 85.5 % agreement and HansenLBS 65 % agreement, while AsnicarF reached 38 % and VilaAV 17 %, so portability of ecotype boundaries is partly a property of the receiving cohort rather than of the framework alone. 3 One caveat limits how much of this gap counts as cross-study instability at all: the consensus labels being compared against were themselves an LDA-plus-Gaussian-mixture consensus at 48.9 % cross-method agreement, so part of the LOSO gap is the baseline disagreement between an LDA projection and that consensus rather than study-to-study variation. 3
The candidate lists show the same split between pooled generation and cross-study survival. E1 Tier-A, the project’s top candidate tier, comprised 51 candidates from a meta-analysis across two sub-studies, HallAB_2017 and NielsenHB_2014 (82 CD / 280 nonIBD), all 100 % sign-concordant and headed by mediterraneibacter gnavus (+4.85), streptococcus salivarius (+3.26), S. thermophilus (+2.69), Erysipelatoclostridium innocuum (+2.65), S. parasanguinis (+2.44) and Enterocloster asparagiformis (+2.41) — described as classical CD pathobionts rather than the ecotype-marker commensals that an earlier within-ecotype differential-abundance pass produced. 3 E3 Tier-A comprised 40 candidates from the single HallAB_2017 study (22 CD / 31 nonIBD) and is flagged provisional pending replication, which requires a second curatedMetagenomicData IBD study populating E3 with both CD and nonIBD samples and was described at an earlier stage of the project as blocked until HMP2 raw data (PENDING_HMP2_RAW) were ingested — an earlier expectation that the project’s later external projection undercuts, recorded as a tension below rather than reconciled. 3 On external projection, 45 of the 51 E1 Tier-A candidates were CD-up in both cohorts, 88.2 % sign-concordance, measured on per-species CD-versus-nonIBD CLR-Δ within HMP2-projected E1 samples comprising 593 CD and 337 nonIBD. 3 Top replicators include M. gnavus (HMP2 effect +1.08, FDR 3e-13), E. asparagiformis (+0.89, FDR 1e-21), H. symbiosa (+1.18, FDR 6e-22), E. innocuum (+0.28, FDR 4e-16), E. bolteae (+1.27, FDR 2e-18) and E. clostridioformis (+1.04, FDR 2e-21). 3 Six Tier-A species are named as actionable — H. hathewayi, mediterraneibacter gnavus, E. coli, E. lenta, flavonifractor plautii and E. bolteae — reported against the same 88.2 % E1 Tier-A external replication in HMP2. 3
E3 could not be put through the same test, and the obstacle is structural rather than biological. The 40 E3 candidates derive from HallAB_2017 only, and the HMP2 projection places just 10 of 130 subjects in E3 against 106 in E1, so HMP2 cannot supply a second-substudy E3 Tier-A meta-analysis; independent E3 replication requires an additional curatedMetagenomicData sub-study carrying both CD and nonIBD samples in E3, and currently only HallAB_2017 qualifies. 3
The project accordingly grades its own cross-cohort replication for this pillar as weak: single-study E3 evidence combined with marginal ecotype stability means the Tier-A list, particularly E3, should be treated as hypotheses for further experimental validation rather than established targets, a limitation the downstream UC Davis cocktail drafts inherit and which the candidate notebooks are instructed to annotate with explicit per-candidate cross-cohort support. 3
Not every signal was cohort-bound. A cohort-meta design over 1,349 samples × 4 cohorts, combining per-cohort Mann-Whitney CD-versus-HC tests by Stouffer’s z, gave meta z = 11.71, p = 1.1e-31 for ebf RPKM (reads per kilobase per million) biosynthetic-gene-cluster abundance and meta z = 11.97, p = 5.1e-33 for ecf RPKM. 3 The same report elsewhere states that both biosynthetic-gene-cluster families are CD-up across the 4 cohorts at p<1e-31; that threshold statement and the detailed ebf p = 1.1e-31 do not sit consistently against each other, and both are recorded here as given rather than reconciled. 3 All four cohorts — HMP2-IBDMDB, MetaHIT, LLDEEP-NLIBD and PRISM — showed CD > HC for both ebf and ecf with cliff-deltas 0.17–0.73, replicating a previously reported immunoactive fatty-acid-amide biosynthetic-gene-cluster family CD-up finding (Elmassry 2025) and giving the largest single effect in that project so far. 3 This refines the concept rather than overturning the metabolomics result: the CD > HC direction held in all four cohorts for gene-abundance features tested within each cohort and combined meta-analytically, with per-cohort cliff-deltas spanning 0.17–0.73 rather than one shared magnitude, whereas pooled absolute-intensity metabolite clustering separated by cohort instead. 3,1
Several of the same project’s conclusions are explicitly scope-limited rather than refuted. The strain-gene analysis used one cohort only (Kumbhari/LSS-PRISM), with the cross-cohort sensitivity test acknowledged as weak. 3 One hypothesis (H3e) has no project-internal replication path at all and would need a multi-cohort meta-analysis to firm its signals up. 3 A site-stratified serology correlation reached r = +0.46 at a single site (Harvard) but was pulled toward null by cohort aggregation, again under a structural single-cohort caveat. 3 The report states that its two PARTIAL verdicts (H3e, H3a-new) reflect single-cohort data-scope caveats, not biological refutation — a distinction this concept requires, since absence of cross-cohort evidence is not evidence of non-portability. 3 The external test cohort is itself small — UC Davis n = 23 patients — so the per-patient cocktail drafts informed by it are explicitly flagged as requiring validation on external CD cohorts before generalization. 3
Feature namespace also changed the projected biological result. The held-out cohort was processed with Kaiju NCBI-NR read classification while the reference embedding was trained on MetaPhlAn3 marker-gene relative abundance, so the two representations were never the same measurement. 1 Latent Dirichlet Allocation projection on pseudo-counts across MetaPhlAn3 and Kaiju namespaces produced plausible Kuehl proportions of 27/42/31% across ecotypes, whereas centered log-ratio (CLR) plus PCA Gaussian-mixture-model projection assigned all 26 Kuehl samples to E3 at confidence greater than 0.97. 1 The source states both that Kuehl detected only 54% of the training species and that 70% of training species had no Kuehl detection; the two statements do not sit consistently against the same denominator and are recorded here as given rather than reconciled. 1 Either way the detection shortfall was large enough that CLR pseudocount imputation drove those sparse vectors into the region of feature space occupied by one dominant Gaussian cluster regardless of actual biology. 1 This supports environmental-resistome and shows that missing feature coverage can create apparently confident but non-portable classifications. 1 The implication drawn is procedural: when projecting across classifier namespaces, use pseudo-count or sparse-data-robust methods such as LDA as primary and treat CLR-based projections as advisory at best. 1 The Web of Microbes–Fitness Browser bridge reinforces this concern from a metabolomics perspective: curated matching linked 19 produced metabolites from Pseudomonas FW300-N2E3 to Fitness Browser substrate experiments, but ambiguous compound matching and a small organism set limited interpretation. 2 Thus, even when a cross-collection bridge exists, its biological portability depends on strain identity and feature-matching quality. 2
Functional annotation coverage adds a related portability constraint. COG (Clusters of Orthologous Groups) analysis across 32 species found that COG annotations covered approximately 70% of genes, while eggnog v6 annotations may differ from original COG assignments. 5 This refines the namespace argument: cross-cohort functional comparisons require harmonized annotation versions and explicit treatment of unassigned genes, not only harmonized taxonomic or compound identifiers. 5 The same analysis found novel or singleton genes enriched in mobile-element and defense categories, including COG L at +10.88% and COG V at +2.83%, whereas core genes were depleted in translation (COG J) by -4.65%; these distributions show that functional representation can also track genome novelty and ecological adaptation rather than a stable portable signal. 5
The plant analysis supports this annotation-schema concern: 12 of 22 marker Pfams were absent from bakta_pfam_domains despite being abundant in InterProScan, and the ten non-zero bakta Pfams returned only 10–35% of InterProScan hit counts. 4 Its use of InterProScan for secretion-system and cell-wall-degrading-enzyme markers therefore illustrates why apparently identical functional classifiers require version- and database-specific validation. 4
A preliminary pooled analysis called C. scindens CD-enriched at log₂FC=+2.67 from pooled Mann-Whitney differential abundance, and the digest interpreted that call as a compositional and stratification artifact — an interpretation argued from proposed mechanisms rather than from a completed compositional-aware test. 1 After leakage repair, C. scindens was genuinely CD-increased under within-substudy analysis, with pooled CLR-Δ +1.18, false discovery rate (FDR)=1e-8, and 4/4 sign concordance; on that reading the leakage-affected within-ecotype call, not the CD-increase itself, is the artifact. 1 The correction supports phenotype-database-coverage-bias by showing that pooled associations can change when leakage and study structure are controlled. 1
The preliminary call’s own diagnosis should be kept separate from the repaired result. C. scindens is described as a well-documented protective species — a secondary bile-acid producer via 7α-dehydroxylation, a TGR5 activator, at ~79 % prevalence in healthy individuals and inhibitory to C. difficile — and three mechanisms were proposed for a false positive: a compositional or relative-abundance artifact in which Ruminococcaceae loss in severe patients inflates the relative abundance of surviving C. scindens strains; strain heterogeneity, since not all strains carry the bai operon so species-level presence is not functional presence; and ecotype mixing, since pooled analysis averages across patient subgroups with different microbiomes. 1 The proposed fix — compositional-aware differential abundance (ANCOM-BC, MaAsLin2 or LinDA) run within patient ecotypes — was a plan at that point, not a completed validation, and its false-call interpretation conflicts with the later within-substudy result recorded above. 1
Independent evidence gating reduced the original 33 within-ecotype Tier-A candidates to 3: Mediterraneibacter gnavus, Flavonifractor plautii, and Blautia wexleri, all in E3; 0/18 E1 candidates passed. 1 This refines classifier interpretation by distinguishing reproducible, within-study-supported biomarkers from candidates generated by pooled compositional structure. 1
Pathway category definitions were themselves non-portable across analytical specifications. Regex-based categories yielded 3 of 52 CD-up pathways in 7 themes and a structurally degenerate failure, whereas a MetaCyc hierarchy assigned 262/409 pathways to at least one of 12 IBD themes and supported iron/heme acquisition with OR=8.1, FDR=7e-6, and 15/52 CD-up pathways. 1 The evidence supports using curator-validated ontology hierarchies as the primary pathway representation and regex categories as sensitivity analyses. 1 Plant-associated pathway inference provides a complementary calibration: GapMind predictions agreed with BacDive metabolite-utilization data at 83.1% consistency, but its core-level completeness score was 0% across compartments, showing that threshold choice can remain non-portable even when two resources broadly agree. 4
The plant marker analysis also supports external validation and continuous rather than categorical interpretation. After stricter marker gating, 878 of 1,115 plant-associated species, or 78.7%, were classified as dual-nature; in a curated panel, all four neutral controls were misclassified, while the continuous pathogenicity ratio distinguished known beneficial from pathogenic species with Mann–Whitney U = 9 and p = 0.027 for N = 7 versus 7. 4 This parallels the IBD result: high apparent class coverage or confidence does not establish biological portability when labels and feature definitions are coarse.
Structural confounding in pooled public cohorts
pitfalls records the design-level reason pooled public cohorts behave this way, which explains rather than merely parallels the classifier failures above. curatedMetagenomicData bundles samples from ~50+ published studies into CMD_HEALTHY and CMD_IBD buckets whose healthy and disease sub-studies do not overlap — every CMD_HEALTHY sample comes from a healthy-cohort study and every CMD_IBD CD sample from an IBD-cohort study — so there is no sub-study that contains both CD and HC samples. 6 Scoped to the 8,489 ecotype-assigned samples of one integrated mart, 45 sub-studies have ≥ 10 HC samples, 5 sub-studies have ≥ 10 CD samples, and 0 sub-studies have both ≥ 10 HC and ≥ 10 CD. 6
The statistical consequence is not a loss of power but a loss of identifiability: a pooled log_abundance ~ diagnosis + (1 | substudy) linear mixed-effects model on the CD-versus-HC contrast is structurally unidentifiable because the substudy random effect perfectly predicts diagnosis, and the reported statsmodels.mixedlm fit with lbfgs failed silently, returning without error but with no usable fixed-effect estimate. 6 The design-consistent alternative is a confound-free within-study contrast: four IBD sub-studies in curatedMetagenomicData — HallAB_2017, LiJ_2014, IjazUZ_2017 and NielsenHB_2014, aggregate 242 CD / 369 nonIBD — hold both CD and nonIBD samples at ≥ 10 each, supporting a within-study CLR-Δ or regression with no study-level confound, combined across sub-studies by inverse-variance weighted meta-analysis. 6 The same reasoning generalizes to any pooled public-dataset analysis, including SRA study inventories and Qiita studies, in which the case and control cohorts were collected by different groups; there too the within-cohort case-versus-control contrast is the design-consistent alternative. 6
Feature keys carry an analogous hazard one level down. Cross-cohort microbiome analysis that pivots on taxon_name_original, or any string key, can silently segregate the same species into two non-overlapping rows when cohorts use different MetaPhlAn3 database vintages or curatedMetagenomicData processing stages, with three such divergences observed in one integrated mart. 6 The remedy adopted in the IBD project supports treating this as shared infrastructure rather than per-analysis cleanup: a synonymy layer built once and joined against by NB00, NB01, NB02, NB03 and every downstream notebook, documented as a project-level pitfall and proposed as a corpus-wide convention, with the NCBI taxid plus GTDB-version-aware rename-table pattern stated to generalize to any project integrating multi-cohort microbiome data. 3
Four figures carry these cross-cohort tests. NB04f_loso_stability.png plots per-substudy leave-one-substudy-out ARI together with per-sample agreement bars, showing the cross-study variation directly. 3 NB04h_hmp2_external_replication.png shows the HMP2 external replication as an ecotype × diagnosis stacked bar, a projection-confidence histogram and an E1 Tier-A scatter of the pooled effect against the HMP2 effect. 3 NB09b_cross_cohort_metabolomics.png is a two-panel figure plotting HMP2 Cliff’s delta against Franzosa Cliff’s delta with themes colour-coded and key metabolites annotated, plus a per-theme sign-concordance bar marking urobilin, acyl-carnitines and long-chain PUFA at ≥75 % concordance. 3 NB09d_metabolite_ecotype_stability.png is a three-panel figure carrying the PCA scatter coloured by metabolite K=4 cluster with PC1 79 % variance identified as cohort batch, a per-cluster diagnosis composition stacked bar at χ²=88, p=7e-17, and a stability summary of cross-cohort LOSO ARI 0.000 against the taxonomic 0.113 baseline. 3
Tensions
Is pooled C. scindens enrichment an artifact or a real within-study signal? One side, the preliminary pooled Mann-Whitney differential abundance and its critique, reads log₂FC = +2.67 as a false call produced by compositional inflation, bai-operon strain heterogeneity or ecotype mixing in a species otherwise documented as protective. 1 The other side, the post-leakage-repair within-substudy analysis, finds C. scindens genuinely CD-increased with pooled CLR-Δ +1.18, FDR = 1e-8 and 4/4 sign concordance. 1 The disagreement is not over the numbers but over whether a within-substudy CD-increase survives compositional-aware differential abundance run within ecotypes, which the corpus proposes but does not report. 1
Did every top-ranked E1 candidate replicate externally? The project states that E1 Tier-A replicates at 88.2 % sign-concordance (45 / 51 candidates CD↑ in both cohorts), including every top-10 candidate and naming S. thermophilus among them. 3 Adjacent text in the same report states that two top-20 candidates fail to replicate — S. thermophilus on effect sign, possibly cohort-specific dairy exposure, and B. stercoris sign-discordant and not significant. 3 Both statements cannot hold for S. thermophilus; the inconsistency is recorded here rather than resolved by preferring either account. 3
How much does LC-MS protocol heterogeneity actually confound cross-study metabolomics? The pooled HMP2–Franzosa comparison attributes near-total cluster separation to batch, with PC1 at 79% of variance and cross-cohort LOSO ARI 0.000. 1 A different analysis in the same digest reports that cross-study LC-MS protocol heterogeneity is effectively absent as a confounder, contrary to the naive expectation for NMDC metabolomics, and that its leucine BQH signal is internally consistent within one analytical protocol — in a 131-sample cohort where 95% of samples (125/131) come from one NMDC study and the remaining 6 samples are from a second study. 1 The two are not averaged here: the first concerns absolute-intensity panels bridged across laboratories and instruments, the second a set almost entirely within one protocol, and the conditions under which protocol heterogeneity does and does not dominate remain untested. 1
What actually blocks independent replication of the 40 E3 Tier-A candidates? One statement in the report describes E3 replication as requiring a second curatedMetagenomicData IBD study that populates E3 with both CD and nonIBD samples, and as blocked until HMP2 raw data (PENDING_HMP2_RAW) are ingested. 3 The later external-projection analysis states that the 40 E3 candidates derive from HallAB_2017 only and that HMP2 places just 10 of 130 subjects in E3 against 106 in E1, so HMP2 cannot provide a second-substudy E3 Tier-A meta-analysis, with only HallAB_2017 currently qualifying. 3 The two accounts cannot both guide action as written — ingesting HMP2 raw data would not by itself populate E3 — and neither is preferred here. 3
Interpretation
The combined evidence indicates that cross-cohort portability is conditional rather than automatic: unitless relative-abundance representations, harmonized feature namespaces, leakage repair, within-study analyses, and independent evidence gating improve interpretability, while absolute intensities, incomplete species detection, and pooled compositional comparisons can produce cohort-specific results. 1 Functional annotation coverage and version differences further refine this interpretation by showing that portability can fail when unassigned genes and schema-dependent categories are treated as equivalent features. 5 Cross-collection metabolomics integration supports extending this interpretation beyond classifier transfer: Web of Microbes action codes distinguish amplified from de novo metabolites, but absent consumption data, ambiguous compound mappings, and a 2018 single-laboratory snapshot constrain transfer to broader metabolic or strain contexts. 2 This conclusion supports multi-omics-integration because microbiome abundance, metabolomics, pathway content, and external validation must be aligned before a cross-cohort biological claim is accepted. 1
Portability should therefore be evaluated at several levels: feature-space compatibility, preprocessing stability, classifier agreement, calibration or confidence behavior, within-study replication, and orthogonal biological corroboration. 1 The findings do not establish that any single normalization method guarantees transfer; rather, they show that representation choice and validation design can materially alter the result. 1 The plant evidence refines this framework by adding label provenance, taxonomic-composition sensitivity, annotation-database coverage, and pathway-threshold calibration as explicit portability checks. 4
Open Directions
- Re-run the metabolomics comparison with ComBat, SVA, RUV, and quantile normalization on the same 122 m/z-bridge metabolites, then test whether cohort separation and LOSO ARI decline without erasing diagnosis-associated structure. 1
- Evaluate external-cohort calibration using the pooled classifier on the UC Davis cohort, testing whether recalibration and a non-constant
is_ibddesign improve the 41% agreement and reduce the 19/22 E1 assignments. 1 - Harmonize MetaPhlAn3 and Kaiju species namespaces, quantify the effect of restoring the training species that Kuehl did not detect, and test whether projected ecotype proportions remain stable across feature spaces. 1
- Reproduce the C. scindens and Tier-A candidate analyses with within-substudy CLR effects, leakage-free splits, and independent cohort replication to determine which biomarkers retain sign and effect size. 1
- Compare curator-validated MetaCyc hierarchy assignments against regex categories in held-out cohorts, asking whether iron/heme acquisition remains associated with IBD when pathway definitions are fixed before testing. 1
- Extend multi-omics-integration by linking strain-level genomic content, metabolite measurements, and abundance across cohorts, while testing whether species-abundance-mediated and strain-content-mediated signals transfer differently. 1
- Re-ingest current Web of Microbes or Northen laboratory data with consumption measurements, then test whether metabolite-production and consumption features transfer across laboratories when compound mappings are restricted to high-confidence identities. 2
- Benchmark COG and eggnog annotation versions across cohorts, measuring whether harmonized functional namespaces and explicit unassigned-gene handling change classifier stability. 5
- Test plant-compartment classifiers with taxonomically balanced resampling, InterProScan/bakta harmonization, and held-out host or metagenomic cohorts to determine whether the refined R² = 0.071 signal transfers beyond the original metadata and database boundaries. 4
- Re-run metabolite-feature clustering on the 326-subject × 111-feature panel after batch correction and without the plan-imposed K=4, asking whether the 0.000 cross-cohort LOSO ARI rises toward the 0.113 taxonomic baseline or whether a natural K=2 HMP2-versus-Franzosa split persists. 3
- Bridge the 9 polyamines by compound-name annotation rather than m/z alone, testing whether the single-cohort OR = 14.6 polyamine enrichment replicates in Franzosa. 3
- Populate E3 with both CD and nonIBD samples from a second curatedMetagenomicData IBD study, so the 40 provisional E3 Tier-A candidates can reach the replication status already shown for 45 of 51 E1 candidates. 3
- Extend the single-cohort strain-gene (Kumbhari/LSS-PRISM) and H3e analyses to a multi-cohort meta design, converting their PARTIAL verdicts from data-scope limits into actual portability tests. 3
- Re-test the ebf/ecf cohort-meta result (4 cohorts; meta z = 11.71 and z = 11.97) in cohorts processed with a different gene-abundance pipeline, isolating whether its transfer owes to the feature type or to the meta-analytic design. 3
- Replace the structurally unidentifiable pooled CD-versus-HC contrast with the within-study CD-versus-nonIBD design across HallAB_2017, LiJ_2014, IjazUZ_2017 and NielsenHB_2014 (aggregate 242 CD / 369 nonIBD), combined by inverse-variance weighted meta-analysis, and report how the resulting effect sizes differ from the pooled estimates they replace. 6
- Model what makes a receiving study align with the consensus ecotypes by regressing per-substudy LOSO ARI and per-sample agreement — spanning LifeLinesDeep at 85.5 % to VilaAV at 17 % — on processing vintage, sequencing depth and cohort composition, and subtract the LDA-versus-consensus baseline disagreement (48.9 % cross-method agreement) so that study-driven instability is separated from method-driven instability. 3