The same query or analysis on the KBase Data Lakehouse does not behave the same way in every execution environment. In the cases the pitfalls digest records, the REST interface hits timeouts and rejects some query text, while direct spark sql handles the same work or accepts the same text. The digest recommends direct Spark as the alternative, but it does not show that direct Spark can never time out or fail. Spark session construction differs between hosted notebooks, cluster-side scripts and local machines. Off-cluster object-store access and Spark Connect daemon recovery each need steps specific to the environment. Together these operational records, collected in the central pitfalls digest (pitfalls), support the working claim that the access path is part of the method and should be reported with it 1.
REST interface versus direct Spark SQL
| REST API error | Reported cause | Reported remedy |
|---|---|---|
| 504 Gateway Timeout | Query took too long | Simplify query, add filters, use direct Spark src: [[summaries/pitfalls |
| 524 Origin Timeout | Server didn’t respond | Retry after a few seconds src: [[summaries/pitfalls |
| 503 “cannot schedule new futures after shutdown” | Spark executor restarting | Wait 30s, retry src: [[summaries/pitfalls |
The table above reproduces the digest’s mapping of REST API status codes to causes and remedies 1.
The digest calls the REST /count endpoint particularly unreliable: it frequently returns errors or times out for tables that Spark queries handle instantly 1.
The REST /schema endpoint frequently times out for large tables. For more reliable schema introspection, the digest recommends running DESCRIBE database.table through the /query endpoint instead 1.
Query text that one path accepts can be rejected by the other. A double hyphen (--) inside a quoted species-clade identifier is not a problem in direct Spark SQL. The REST API, however, rejects any query containing -- regardless of quoting, treating it as a SQL comment metacharacter, and returns a 400 error 1. This refines the general contrast between REST and Spark: the difference is not only about scale and timeouts but also about which query strings are accepted at all 1.
The digest recommends direct spark.sql() on the cluster when a query involves >1M rows, JOINs across large tables or aggregations on billion-row tables, or when the REST API keeps timing out 1.
Session construction differs by environment
The digest distinguishes three environments that use different import patterns to obtain a Spark session, and warns that using the wrong one causes ImportError 1.
- JupyterHub notebooks:
spark = get_spark_session()works with no import, because/configs/ipython_startup/00-notebookutils.pyinjects the function 1. - Regular Python scripts on the cluster: these need the explicit import
from berdl_notebook_utils.setup_spark_session import get_spark_session, from the same module the notebooks use. The fitness_modules project found that this import works from regular Python scripts, not just notebooks 1. - Local machine: the listed import is
from get_spark_session import get_spark_session, which differs from the cluster-script import 1. - Failure signature: calling bare
get_spark_session()(no import) in a CLI script on JupyterHub raisesNameError, because the auto-import only applies to notebook kernels 1.
Off-cluster access and daemon recovery
When the MinIO client (mc) is used off-cluster, proxy environment variables must be set or commands time out 1.
The conservation_vs_fitness project reported that the Spark Connect service runs as a Java process on port 15002. Killing Java processes, for example while cleaning up stale notebook processes, takes down Spark Connect, after which get_spark_session() fails with RETRIES_EXCEEDED / Connection refused 1.
The reported recovery is to log out of JupyterHub, start a new session and run get_spark_session() from a notebook, which restarts the Spark Connect daemon. The daemon cannot be restarted from the CLI 1.
The ingest skill’s initialize() is off-cluster-only. On JupyterHub, the digest advises bypassing initialize() and building Spark and MinIO clients directly 1.
Evidence strength
These are operational observations from a cross-project pitfalls digest, several of them attributed to single projects. They are not systematic measurements of failure rates. The rule that the access path should be treated as part of the method is therefore well supported in practice but not backed by a quantified benchmark. The evidence also does not show that direct Spark is immune to timeouts; it only shows that the digest recommends direct Spark where REST fails 1. Related failure classes are covered in silent-failure-modes-in-distributed-queries, long-running-analysis-execution-reliability and driver-side-result-collection-limits.
Open Directions
- Run the same table counts and
DESCRIBEcalls through REST/count,/schemaand/queryand through direct Spark SQL, on tables of increasing size, and log the status codes and timeouts for each path. Measured failure rates per path would replace the qualitative description “frequently times out” and would test whether direct Spark also fails at scale 1. - Add a check before submission that flags
--in query text bound for the REST API, and record in each project’s provenance which access path it used, so results can be traced to the environment that produced them 1. - Provide a single session helper that detects whether it is running in a notebook, a cluster script or on a local machine, and test it against the
ImportErrorandNameErrorfailure signatures the digest lists 1.