Distributed Computation and Analysis Execution

This hub covers how analyses on the KBase Data Lakehouse actually run. It asks where queries execute, how results move from the spark sql cluster into a single Python process, which layouts and code patterns make large genomic jobs slow, and which failures stop work, sometimes without an error. Here, Spark is the distributed query engine behind the lakehouse, and the driver is the single process that collects its results. The corpus addresses these questions through one central source, the cross-project pitfalls digest. That digest records operational incidents, workarounds and a few timings from named analyses. It does not provide systematic benchmarks or failure rates ₁. The theme running through those records is that execution conditions belong to the method. Access path, session lifetime, result collection and compute budgets can each change whether an analysis finishes and what it returns ₁.

Literature Context

Published work frames Apache Spark as a fast, fault-tolerant engine for genomic data. A survey of Spark bioinformatics applications describes an in-memory, iterative framework whose resilient distributed dataset abstraction provides “high fault tolerance and high scalability.” It reports that Spark can be up to 100 times faster than Hadoop for memory access and 10 times faster for disk access, and it lists Spark SQL for structured data. These are survey-level claims, not new benchmarks PMID 30101283. A sequence-clustering tool reports near-linear scaling with input size and node count, with similar performance on cloud and high-performance computing environments PMID 30816928. Several papers treat data layout and load balance as the main levers. A bin-based index for genomic range joins credits balanced partitioning for its scalability. Against previously available tools, it reports up to 933.5x speedup with its Message Passing Interface implementation and up to 4.65x with its Spark add-in PMID 37022216. A Spark variant caller tackles computation skew with adaptive data segmentation. On the authors’ own cluster, at the best performance achievable there, it was 74% faster than GATK3.8 HaplotypeCaller on a single node, and 57% faster than GATK4.0 HaplotypeCallerSpark and 27% faster than SparkGA on multiple nodes PMID 30764760. A bisulfite aligner likewise redistributes imbalanced data to reduce delays PMID 30526492.

Memory-bounded design is a second theme. REGENIE loads only local segments of the genotype matrix rather than genome-wide matrices, which saves compute time and memory; it was demonstrated on up to 407,746 UK Biobank individuals PMID 34017140. Earlier grid-computing work argued that large, dispersed data may require moving computation to the data. The same work named authentication, authorisation and audit systems as poorly scalable bottlenecks for distributed collaboration PMID 17000735.

This hub’s findings are mostly consistent with that literature, though they come from incidents recorded in the pitfalls digest rather than from controlled benchmarks. The digest’s rule to keep relational work inside the engine and avoid driver-side collection (driver-side-result-collection-limits) parallels REGENIE’s segment-wise loading PMID 34017140. It also parallels the principle of moving computation to the data PMID 17000735. Its working hypothesis that unpartitioned billion-row tables force full scans (join-strategy-and-table-partitioning-cost) agrees with the partitioning and skew emphasis of the range-join and variant-calling work PMID 37022216, PMID 30764760. The corpus’s only measured broadcast change gave an 8% improvement. Those papers report large algorithmic speedups, so this small gain extends the literature with a cautionary data point rather than contradicting it. The credential and tenant failures in credential-and-tenant-access-lifecycle echo the authorisation bottleneck named in grid-computing work PMID 17000735.

The corpus also sits in tension with how the literature presents Spark. The survey stresses fault tolerance and scalability PMID 30101283. The digest, by contrast, documents temporary views lost on reconnect, queries returning 0 rows without an error, 30+ minute jobs killed without a message, and access-path-dependent REST errors (silent-failure-modes-in-distributed-queries, long-running-analysis-execution-reliability). None of the supplied abstracts report such session- or platform-level failure modes or their rates. Only abstracts were available, so the full papers may differ. The corpus gives no rates either, so how often these failures occur remains unquantified.

What the Corpus Shows

The access path is part of the method. The same work behaves differently depending on how it reaches the lakehouse. Through the REST (web) interface, the digest reports 504 and 524 timeouts and 503 errors while Spark executors restart ₁. The /count endpoint frequently returns errors or times out for tables that Spark queries handle instantly. The /schema endpoint frequently times out for large tables, and the digest recommends DESCRIBE through /query instead ₁. The REST API also rejects any query containing -- regardless of quoting, which direct Spark SQL accepts ₁. Session construction differs again across JupyterHub notebooks, cluster scripts and local machines. Using the wrong import raises ImportError or NameError ₁. Access preconditions sit beneath all of this. Error 20040 from get_group_sql_warehouse(tenant_name) signals missing tenant-group membership. An expired KBASE_AUTH_TOKEN makes every API call return 401/403 ₁. These records support recording, for each project, which path and environment produced a result, as argued in execution-environment-dependent-access-paths and credential-and-tenant-access-lifecycle ₁.

Keep relational work inside the engine. Three member pages reach the same design rule from different failures. In driver-side-result-collection-limits, a join over 1B-row tables, filtered to >200K elements via broadcast (a join strategy that sends the small-side table to every worker), hit spark.driver.maxResultSize = 1024 MB when collected with .toPandas() ₁. In a separate case, a pandas spatial merge built a working set of about 9 GB and ran out of memory on a 16 GB driver ₁. A donor-inference step that exploded data into a roughly 126M-row intermediate also ran out of memory. Rewritten as a join plus subtraction, the same step ran in 142s ₁. numeric-type-fidelity-across-engine-boundaries supports the same rule from the type side. Spark DECIMAL values arrive as decimal.Decimal and raise TypeError in float arithmetic. Arrow-backed frames from Spark Connect, Spark’s client–server interface, fail when passed back to spark.createDataFrame(). The remedy given is to avoid the pandas-to-Spark round trip ₁. vectorization-versus-row-wise-iteration extends the theme to code run after collection. An iterrows() loop over 27K focal features × 218K contigs ran 30+ minutes without completing, while a vectorized pd.merge finished in ~10 seconds ₁.

Physical layout appears to set the cost of billion-row joins. join-strategy-and-table-partitioning-cost proposes the working hypothesis that table partitioning is a major cost driver. Joining gene_genecluster_junction (~1B rows) with gene (~1B rows) takes 3-5 minutes per species; neither table is partitioned by gene_cluster_id or genome_id, so every query requires a full table scan ₁. On Smeli, broadcast hints (explicit requests to send the small filter tables to every worker) gave ~274s against ~300s, an 8% improvement. A two-stage rewrite took ~310s, no improvement. Collecting 1-2M rows took <2s ₁. This refines the collection theme: here the scan was the bottleneck, not the driver. When the filter matches the partition key, as with gene_cluster partitioned by gtdb_species_clade_id, per-clade queries are fast. In the same setting, LIMIT/OFFSET pagination was orders of magnitude slower ₁. The proposed repartitioning of the large tables has not been measured ₁.

Failures are often silent. silent-failure-modes-in-distributed-queries documents results that look valid but are empty. A temporary view lost when Spark Connect reconnected made later joins return 0 rows with no error. Cold databases can return a valid /count alongside empty /query results ₁. Permission failures can appear as storage or catalog errors such as 403 Forbidden or AccessControlException ₁. long-running-analysis-execution-reliability adds kernels idle-timed out after ~17–25 minutes. Spark jobs taking 30+ min were killed with no error message ₁. The workarounds are standalone nohup scripts, intermediate Parquet files (a columnar on-disk table format) at each stage and _finalize.py recovery scripts. They make partial results recoverable but do not guarantee that a run completes ₁. Token expiry mid-run is a further cause of failures that surface late ₁.

Some computation is deliberately approximate. runtime-budgeted-resampling-and-embedding records budget choices as method choices. The digest recommends bootstrap sizes of 250-400 with fixed seeds. Bootstrap here means resampling with replacement to estimate confidence intervals ₁. A cosine-metric umap run (UMAP is a nonlinear dimension-reduction method) on 83K genomes with 64 dimensions took >60 minutes and did not complete on a single-CPU JupyterHub pod ₁. It ran in ~7 min after L2 normalization (scaling each vector to unit length), a Euclidean metric and fitting on a 20K subsample ₁. The metric swap rests on an exact identity. The subsample fit is an approximation whose effect on the layout is unmeasured ₁.

Tensions and Caveats

• Single source, no rates. Every member page rests on the pitfalls digest. That makes the guidance well-documented practice, not a quantified finding. The digest gives no frequency for token expiry, view loss, REST timeouts or idle kills ₁.
• Broadcast handling. Disabling auto-broadcast (spark.sql.autoBroadcastJoinThreshold = -1) was blamed for a 17+ minute hang on a 13.7M × 18K join. Explicit broadcast hints gave only an 8% improvement for Smeli. The digest does not reconcile the two, and no rerun with the default setting restored is reported ₁.
• The maxResultSize setting. One entry says the property is read-only after session start and cannot be raised with SET. Another suggests raising it in the session config. Whether Spark Connect users can set it at session creation is unresolved ₁.
• The asserted speedup. The digest claims a 1000× vectorization speedup “consistent across pandas use cases.” Its own cases give one incomplete row-wise run and one “seconds instead of minutes.” The direction of the effect is documented; its size is not ₁.
• Direct Spark is not shown immune. The digest recommends direct Spark where REST fails, but it does not show that Spark itself never times out ₁.
• Untested remedies. Repartitioning, batching the spatial merge for polysaccharide-utilization loci (PUL, clustered polysaccharide-degradation genes) and pushing its filter into Spark, and the default-broadcast fix are proposals without reported outcomes. Whether nbconvert headless runs avoid idle termination is also untested ₁.

Where to Go Deeper

• driver-side-result-collection-limits — read first. It holds the most concrete failure sizes, and its algebraic rewrite is the only remedy on that page with a reported runtime ₁.
• join-strategy-and-table-partitioning-cost — read next to separate scan-bound slowness from collection failures, and for the broadcast tension ₁.
• silent-failure-modes-in-distributed-queries — for the checks to run before treating an empty result as a finding ₁.
• long-running-analysis-execution-reliability — for the checkpoint and nohup pattern used for jobs that outlast notebook kernels ₁.
• execution-environment-dependent-access-paths — for the comparison of REST and direct Spark, and for session setup in each environment ₁.
• credential-and-tenant-access-lifecycle — for pre-flight checks that separate token expiry from missing tenant membership ₁.
• numeric-type-fidelity-across-engine-boundaries and vectorization-versus-row-wise-iteration — for defensive coding patterns after collection ₁.
• runtime-budgeted-resampling-and-embedding — for which settings to report when bootstrap or UMAP shortcuts are used ₁.

Key entities: spark sql, umap, kescience-fitnessbrowser, bacteroidota.

Report: pitfalls.