The BERDL Data Atlas — Inventory, Topic Map, and Cross-Reference Synergies shows that biological data availability in the KBase Data Lakehouse is structured not only by scientific topic but also by ownership, tenant boundaries, and the uneven reuse of cross-tenant links. 1 The atlas inventories 1,740 deduplicated tables across 119 databases, 17 tenants, and 10 funding agencies or programs, covering 17 biological topics. 1
Ownership concentration produces coverage bias
DOE accounts for approximately 78% of the KBase Data Lakehouse tables: DOE-BER contributes 63%, DOE BRaVE 14%, DOE/NSF 0.6%, and DOE-FE 0.4%; ARPA-H contributes 4% and NSF contributes 3.5%. 1 DOE-BER is the only agency covering all 15 biological topics recorded in the agency-coverage analysis. 1 Six topics are more than 75% single-owner, including mobile_phage at 96% PhageFoundry ownership, pangenome at 79% KBase ownership and reference_protein at 78% refdata ownership, while 7 topics are cross-tenant with a top-tenant share of at most 55%; taxonomy spans 12 tenants and is the broadest cross-tenant surface. 1 The same topic-coverage finding describes Defense / HHS as mono-topic through PhageFoundry (all mobile_phage), an attribution that conflicts with the user-confirmed DOE BRaVE mapping for phagefoundry recorded in the atlas caveats (see Tensions). 1
This ownership pattern means that apparent topic availability can reflect the location and funding history of data rather than a balanced distribution of independent observations. 1 The field_observational topic represents 40% of tables, mobile_phage 14%, fitness_phenotype 11%, and genome 6.4%, while all other primary topics represent less than 4% each. 1 Sixteen tables, or 0.9%, remain unclassified and consist of personal scratch or one-off survey data. 1
The pitfalls supports treating ownership and provenance as coverage variables: 83,227 of 293,059 genomes, or 28.4%, have AlphaEarth environmental embeddings, while 52.7% of 180,025 genomes have unknown per-genome NCBI environment labels. 2 These gaps can reflect missing coordinates or unstructured metadata rather than biological absence, so owner- and tenant-specific coverage should not be interpreted as population absence. 2
Tenant breadth is uneven
Taxonomy spans 12 tenants and has the broadest cross-tenant surface, whereas several other topics are concentrated within individual tenants. 1 KBase covers 10 topics with entropy 2.87 and is described as the most evenly cross-topic tenant; NMDC covers 11 topics with entropy 2.61; PROTECT covers 6 topics with entropy 2.37; KEScience covers 11 topics with entropy 1.83; and ENIGMA covers 5 topics with entropy 0.43 and is described as deep-but-narrow; PhageFoundry and USGS are mono-topic. 1 ARPA-H (PROTECT) spans 6 topics, while NSF (Planet Microbe), DOE-FE (NETL) and DOI (USGS) are narrow in topic but cover unique sample types. 1
These distributions support a distinction between data volume and portfolio breadth: a tenant may contain many tables while still being concentrated in a small number of topics. 1 ENIGMA contains 36% of tables but appears in 6 audited projects, while PhageFoundry contains 14% of tables but appears in 5 projects and PROTECT contains 4% but appears in 2 projects. 1
The pitfalls evidence refines this distinction by showing that tenant ownership is also operational: KBase Data Lakehouse’s migration between Delta and Iceberg changes live table addresses, and in a data_lakehouse_ingest config the tenant field is the MinIO governance group name rather than the database name prefix, with the library auto-prepending the tenant name to the dataset field to form the namespace. 2 Queries must therefore discover the live catalog and resolve tenant and database names before comparing apparent breadth or concluding that a resource is unavailable. 2
Ownership affects realized integration
Among 66 audited BERIL projects, 51, or 77%, span multiple tenants. 1 KBase appears in 53/66 projects, or 80%, and KEScience appears in 35/66, or 53%. 1 The realized kbase × kescience bridge accounts for 36 cross-tenant projects, mostly pangenome-by-fitness joins through genome_id and ncbi_taxon_id. 1 Heavy presence in the lakehouse does not mean heavy reuse: ENIGMA holds 36% of tables but appears in only 6 projects, PhageFoundry holds 14% of tables and appears in 5 projects, and PROTECT holds 4% and appears in 2 projects. 1 The report interprets the kbase × kescience dominance as following from both tenants being intra-DOE-BER, well documented and covering the most cross-tenant genome-by-phenotype topics; this is the authors’ interpretation rather than a tested causal account, and the five untapped use cases UC1–UC5 are framed as the next class of analyses, extending into refdata, nmdc_arkin, PhageFoundry/PROTECT and ENIGMA. 1
This realized-use pattern supports cross-tenant-data-bridging: ownership boundaries do not prevent integration when shared identifiers and operational demand connect tenants, but they can make some scientific combinations more visible than others. 1 The atlas identifies 536 unordered cross-tenant bridges at tenant-by-topic-cell granularity from 29 canonical join keys, but only UC1 was sample-validated in the study. 1
The pitfalls document supports the need for that value-space validation: in one pangenome linkage example, 12 of 32 joins through ncbi_strain_identifiers were incorrect genus matches, including an MT20 collision between Rhodanobacter glycinis and Streptococcus pneumoniae. 2 It also refines the bridge audit by showing that NMDC classifier and metabolomics file_id namespaces do not overlap and require a sample_id bridge through omics_files_table. 2
The contrast between possible and realized integration also refines cross-tenant-data-bridging: five high-leverage bridges had zero realized use at audit time, including proposed links between ENIGMA and PhageFoundry, NMDC and PROTECT, and NMDC and refdata. 1 Schema-level compatibility does not establish value-space overlap because identifiers such as genome_id can represent different identifier systems in KBase, NCBI, and metagenome-assembled genome (MAG) pipelines. 1
Name-based discovery hides ownership
The atlas’s table_topic_map.csv is described as the first comprehensive, machine-readable catalog of the KBase Data Lakehouse with tenant, agency, program and biological-topic provenance; its composite figure nb04_atlas_composite.png combines a tenant × topic heat map, a tenant reuse bar and a theory-vs-practice scatter, and nb05_volume_by_entity_class.png shows per-entity-class data volume on a log scale with exact counts. 1
The NMDC Context Audit refines this ownership picture by showing that database names do not reliably encode provenance: the 20 database names containing “nmdc” resolve to 7 real, maintained resources across three tenants (nmdc, kbase, plus broken user copies), falling into six provenance classes. 3
- Genuine NMDC:
nmdc.metadataandnmdc.results, from the NMDC program (DOE-BER). 3 - External re-host (NCBI):
nmdc.ncbi_biosamples. 3 - External re-host (Pfam):
nmdc.ref_data, from the Pfam consortium. 3 - Other-group derivation:
kbase.nmdc_arkin, from the LBNL Arkin Lab. 3 - NMDC-derived, kbase tenant:
kbase.nmdc_mags(KBase / NMDC). 3 - Namesake collision:
kbase.nmdc_neonis NEON (NSF National Ecological Observatory Network), a different program, not NMDC — an acronym collision that would corrupt agency attribution. 3
Two of the four databases in the nmdc tenant are external data re-hosted rather than NMDC output: nmdc.ncbi_biosamples, an NCBI BioSample harvest of 51,711,888 biosamples and 756,112,544 attribute rows, and nmdc.ref_data (Pfam, 27,481 terms); meanwhile three NMDC-related databases (nmdc_arkin, nmdc_mags, nmdc_neon) live in kbase and, because the inventory groups by catalog prefix, are invisible to anyone browsing the nmdc tenant. 3 The genuine NMDC biosample universe is 16,640 samples (nmdc.metadata.biosample_set), while the co-hosted NCBI mirror is 51.7M — a ~3,000× difference with nothing in the names to signal it. 3 NMDC-derived data is split across nmdc.* and kbase.nmdc_* with no cross-link, and the freshest NMDC resource, kbase.nmdc_mags with 62,346 MAGs, sits in the tenant a user is least likely to search for NMDC. 3
NMDC is the DOE-BER National Microbiome Data Collaborative, whose LinkML-based data model defines the *_set schema in nmdc.metadata; NEON is the NSF National Ecological Observatory Network, with an agency and sampling design distinct from NMDC; and NCBI BioSample and Pfam/InterPro are the true authorities for nmdc.ncbi_biosamples and nmdc.ref_data respectively. 3
The audit also refines how ownership bias should be handled: co-hosting is often intentional and valuable, because the NCBI mirror adds an attribute-harmonization layer that makes 51.7M raw NCBI samples analytically usable and the Arkin derivative adds embeddings and traits that do not exist upstream, so the recommended fix is not to relabel or remove resources but to surface provenance and value together. 3
Organism- and resource-level coverage gaps
Single-organism checks show that table-level inventory breadth does not guarantee coverage of a specific organism. 4,5,6 Acinetobacter baylyi ADP1 was checked against the kescience_fitnessbrowser organism table and not found (ADP1 → Fitness Browser: not present), so the explorer’s mutant growth fitness on 8 carbon sources is described as the only source of condition-specific fitness data for this organism in the lakehouse. 4
Escherichia coli genomes are completely absent from GapMind in the KBase pangenome collection: E. coli K-12 (Keio, GCF_000005845.2) had 0 GapMind predictions in the coverage check. 5
The ENIGMA CORAL database (47 tables, enigma_coral) contains no Desulfovibrio vulgaris Hildenborough (DvH) records: its single TnSeq (transposon-insertion sequencing) library is for FW300-N2E2 (Pseudomonas), its DubSeq (dual-barcoded shotgun expression library sequencing) libraries cover E. coli, P. putida and B. thetaiotaomicron, and all 6,705 genomes and 15,015 genes are environmental isolates from Oak Ridge. 6 It does contain 4,346 field samples with geochemistry data (uranium, metals) across 596 Oak Ridge locations and 213,044 ASVs (amplicon sequence variants) for community composition, potentially useful for future analyses but not for gene-level fitness analysis. 6 This supports the atlas’s characterization of ENIGMA as deep-but-narrow. 6,1
The 2018 Web of Microbes (WoM) snapshot contains 37 organisms across 5 ENIGMA-funded projects: 6 biocrust isolates (5,604 observations in BG11 media), 10 ENIGMA groundwater isolates (1,050 observations in R2A medium), 4 other isolates (E. coli, Zymomonas, Synechococcus, Microcoleus), 10 triculture time-series entries and 2 native microbiome entries, and 4 theoretical auxotroph predictions and 1 control. 7
The WoM snapshot records only what organisms produce, not what they consume, which prevents testing whether consumed metabolites predict gene essentiality; it is also small (37 organisms, 5 projects) and from a single laboratory, and 37 organisms (20 experimental) are too few for statistical analyses across organism groups. 7
Some resources sit in user-owned or less-discovered databases: arkinlab_microbeatlas holds 464K global 16S samples, useful as a prevalence/biogeography baseline; protect_integration is a second Protect-tenant database beyond protect_genomedepot; and u_kazakov__klebsiella_genomedepot and u_kazakov__strain_modelling are user-owned Klebsiella phage-host work worth checking with the owner. 8 Separately, the PhageFoundry phagefoundry_strain_modelling database — distinct from the user-owned u_kazakov__strain_modelling collection — was previously overlooked (documented as “timed out during discovery”) and contains 18 tables with experimental phage-host phenotype data. 9
Implications for biological inference
Ownership concentration should be treated as a coverage-bias variable when comparing topics, tenants, agencies, or biological systems. 1 In particular, the 79% KBase ownership of pangenome data and 96% PhageFoundry ownership of mobile_phage data indicate that conclusions about these topics may be strongly coupled to the methods, sampling frames, and curation practices of their dominant owners. 1
The atlas therefore supports data-landscape-ownership-and-coverage-bias as a provenance-aware complement to pangenome-integration and provenance-aware-resource-discovery. 1 Data discovery should record both the biological topic and the tenant or agency distribution, while cross-tenant analyses should validate identifier value spaces before interpreting apparent coverage or association patterns. 1 The pitfalls evidence supports this workflow requirement because archived SQL may use obsolete namespace forms, large tables require live schema inspection and key-filtered joins, and missing metadata or failed joins can produce incomplete coverage without indicating biological absence. 2
Boundary conditions
The realized-use audit mined project README files, so data-source mentions in research plans or notebook source may have been missed and the reported tenant breadth is therefore a lower bound. 1 Two tenant-to-agency mappings, evaluation and lambda, remain unverified by program documentation and account for 4 tables total. 1 phagefoundry and msyscolo were originally inferred as “likely” Defense/HHS and NSF/USDA; user-confirmed corrections to DOE BRaVE and DOE/NSF are folded into the curated 15-row data/tenant_to_agency.csv tenant-to-agency, program and primary-funder map, and verifying evaluation and lambda with program documentation is listed as a remaining step. 1 Across-tenant deduplication was not performed, so apparent inventory differences may include duplicated records across refdata, KBase, ENIGMA, and genome-depot layers. 1
The pitfalls document refines these boundaries by warning that schemas, namespace availability, permissions, API behavior, and naming conventions can change; archived reports and notebooks are historical records rather than automatically valid executable instructions. 2
Tensions
The atlas text carries two agency attributions for PhageFoundry: its topic-coverage finding lists Defense / HHS as mono-topic through PhageFoundry (all mobile_phage), whereas its caveats state that phagefoundry was only inferred as “likely” Defense/HHS and that a user-confirmed correction to DOE BRaVE has been folded into the tenant-to-agency map. 1 Which mapping each agency-level figure was computed from is not resolved here. 1
The NMDC Context Audit contradicts part of the atlas’s provenance labelling: the atlas tags Rhea/GO reference ontologies under nmdc_arkin as “NMDC integrated,” which the audit describes as provenance blur even where documentation exists. 3 Combined with prefix-based grouping that files nmdc_neon and other NMDC-related databases under kbase, this suggests the hypothesis that some atlas tenant- and agency-level shares misattribute externally sourced resources. 3,1
Open Directions
- Re-audit all 66 BERIL projects using README files, notebooks, research plans, and workflow metadata to test whether the observed tenant breadth remains a lower bound and whether underused owners contribute more realized data sources than reported. 1
- Recompute agency and tenant concentration after deduplicating shared records across refdata, KBase, ENIGMA, and genome-depot layers to determine how much of the apparent ownership imbalance reflects duplicated inventory. 1
- Execute UC2–UC5 with value-space validation for their proposed join keys to test whether ownership-separated datasets provide usable biological coverage rather than only schema-level bridges. 1
- Compare topic-level scientific conclusions with and without tenant or agency-stratified sampling to measure whether dominant owners alter observed associations, taxonomic coverage, or phenotype distributions. 1
- Verify the evaluation and lambda agency mappings against program documentation and assess whether correcting the 4 affected tables changes agency concentration estimates. 1
- Recalculate coverage and cross-tenant bridge rates after live catalog discovery, explicit identifier reconciliation, and filtering of missing environmental metadata to distinguish ownership effects from namespace, metadata, and join failures. 2
- Re-run the atlas’s tenant and agency share tallies after reclassifying the six NMDC provenance classes (e.g.
kbase.nmdc_neonas NSF NEON,nmdc.ncbi_biosamplesas NCBI) to measure how much catalog-prefix grouping shifts ownership shares. 3,1 - Recompute agency coverage with PhageFoundry explicitly mapped to DOE BRaVE to settle whether the Defense / HHS mono-topic attribution survives in any reported figure. 1
- Build an organism-by-resource presence matrix (Fitness Browser, GapMind, ENIGMA CORAL, Web of Microbes) for model organisms such as ADP1, E. coli K-12 and DvH to locate systematic organism-level gaps hidden by table-level inventory breadth. 4,5,6,7