Data Engineering · 4 min read

Google Cloud runs BigQuery, Spark and Trino on one Iceberg table

A hands-on Google Cloud Lakehouse test runs BigQuery, Spark and Trino on one Iceberg table with zero sync jobs. Trino beat Spark in three straight commit collisions.

By Grace Kim814 words

Key takeaways

  1. BigQuery, Spark and Trino wrote to one Apache Iceberg table via Google's Lakehouse runtime catalog with no copies or sync jobs.
  2. Trino won 3 out of 3 concurrent-commit collisions against a 30-second Spark MERGE under serializable isolation; snapshot isolation let both commit.
  3. BigQuery DML on Iceberg tables is in Preview and requires copy-on-write write modes.
  4. The 2.3-debian12 Dataproc image ships Trino 432 with Iceberg 1.4.1, requiring a Python auth shim; Trino 481+ drops it.
  5. Iceberg defaults to 4 commit retries with 100 ms to 60 s backoff (commit.retry.num-retries).
One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy - KDnuggets
One Table, Many Engines: Running BigQuery, Spark, and Trino on a Single Iceberg Copy - KDnuggets — mwitt1337 / pixabay

One Apache Iceberg table, one Cloud Storage bucket, one catalog — and three query engines writing to it without a single sync job. A hands-on walkthrough published on KDnuggets (primary contributor: Shittu Olumide) tests Google Cloud's borderless Lakehouse runtime catalog with BigQuery, Apache Spark and Trino, and the results are worth a data team's afternoon.

The setup it replaces is the familiar mess: BigQuery for analytics, Spark for pipelines, Trino for Friday-afternoon ad hoc queries, each with its own copy of the data. Sync jobs keep them aligned, sort of. BigQuery says an order shipped; the Spark copy says paid and awaiting shipment; nobody trusts either until the sync finishes — assuming it finishes, at which point you're paying engineering time to trace the breakage.

How does the shared table actually work?

Every engine reads the same Parquet files and metadata layout. The catalog holds no data, just a pointer to the table's current version via its metadata.json file. Spark and Trino connect through the Iceberg REST catalog endpoint at biglake.googleapis.com; BigQuery reads the same table with a four-part P.C.N.T. name (project.catalog.namespace.table) and needs no connector or connection resource.

The author chose credential vending mode, so only the catalog's own service account touches the bucket, and the catalog hands engines short-lived tokens scoped to the table's paths. Catalog creation is one gcloud command. One gotcha: the service account is created asynchronously, so the IAM grant can fail if run immediately — wait 20 seconds and retry.

What happened in the test?

Spark (on Dataproc's Managed Service for Apache Spark, image 2.-debian12 with optional ICEBERG and TRINO components) created and loaded a five-row orders table in under 10 seconds of actual work. Trino appended three rows. BigQuery ran an UPDATE and a MERGE, each affecting two rows in about four seconds, committing directly to the shared table. Trino's next SELECT showed all nine rows. Snapshot history confirmed four commits: two appends, two BigQuery overwrites. Nothing synced. Nothing was copied.

The table's write modes matter. Copy-on-write is mandatory for DELETE, UPDATE and MERGE, because BigQuery DML on Iceberg tables (currently in Preview) and Google's automatic table maintenance require it. Open-source Trino writes row-level changes as merge-on-read, so Trino sticks to INSERT and SELECT. Row-level changes belong to Spark and BigQuery. For an orders table, that split is natural anyway: new orders append, status changes upsert.

What broke, and what happens when writes collide?

The 2.3-debian12 image ships Trino 432 with a bundled Iceberg 1.4.1, which can't talk to the Lakehouse REST catalog natively or use vended storage tokens. The workaround is a sub-100-line Python shim on each node that adds the Google access token and required headers, forwarding requests to biglake.googleapis.com. Treat it as a stopgap; Trino 481 or later drops both the shim and the extra bucket grant.

Then the stress test. Iceberg uses optimistic concurrency: writers build a new metadata file and ask the catalog to swap the pointer, and a competing commit forces a refresh-and-retry. With strict conflict detection enabled by default, the author ran a deliberately slow 30-second Spark MERGE while Trino inserted two rows 20 seconds in. Trino won every round — three out of three. Spark aborted each time with a ValidationException on the conflicting file, because its serializable-isolation MERGE scanned the whole table and any new file could match.

Switching the table to snapshot isolation (write.merge.isolation-level = 'snapshot') fixed it: both writes committed, with the trade-off that rows appended after the MERGE's scan stay pending until the next run. Meanwhile, a BigQuery reader polling every few seconds never saw a half-written state across all rounds. Every result matched a committed snapshot.

What should a data team take away?

The author's production checklist:

  • Choose isolation per table: serializable for balances and inventory counts, snapshot for status upserts against a table others append to all day.
  • Narrow what a MERGE reads — partition by order date and add it to the MERGE condition, so a fresh append stops blocking last week's rewrite.
  • Tune retries: Iceberg defaults to four pointer-swap retries with a 100 ms-to-60 s backoff; streaming appends under long MERGEs need more.
  • Keep long rewrites away from busy appenders — a 30-second MERGE lost three times in a row to two-second inserts. That's scheduling, not configuration.
  • Know when one table isn't the answer: hard isolation between teams or cross-region requirements still warrant a copy with a clear owner.

The verdict from the snapshot log: three engines, one table, no data lost, no reader confused. The Trino shim is the asterisk, and it has an expiry date.

biglake.googleapis.com (Original)

Price tags

Share this article:

More from Grace Kim

Grace Kim

Show full bio

Market editor covering companies, deals and strategy in Analytics Digest is a professional daily at Analytics Digest.

18 articles

Read also

« Previous articleNext article »