DuckCon #7
Making Iceberg walk and talk like Postgres
Marco Slot · DuckCon #7
6
sections
9
slides
01 / Intro
- A 2.5-year project started at Crunchy Data, now continued at Snowflake, presented at DuckCon #7.
- Goal: make Apache Iceberg feel like a fully native table format inside Postgres.
- Three motivations: bottomless storage that never runs out of disk, fast analytical queries powered by DuckDB under the hood, and transactions spanning both Postgres and Iceberg tables.
- That cross-format transactionality is the standout feature: it lets you move data between Postgres and other systems atomically.
More detail
The speaker frames pg_lake as an answer to three needs at once: infinite object-storage-backed capacity, analytical speed, and transactional data movement across systems.
02 / What it is
- pg_lake is a Postgres extension that adds two capabilities: Iceberg tables in Postgres, and query/import/export of files in object storage.
- Headline benefits: bottomless storage, fast analytical queries, and transactional data pipelines.
- Open source at github.com/Snowflake-Labs/pg_lake.
More detail
Positioned as a lightweight extension rather than a separate engine: you keep using Postgres, but gain object-storage-backed Iceberg tables.
03 / How queries work
- Usage is simple: CREATE TABLE ... USING iceberg, then INSERT and SELECT like any Postgres table.
- On write, Postgres emits the Iceberg files (Parquet data, JSON/Avro metadata) and caches all the metadata in Postgres tables to avoid re-reading it.
- On query, the whole query or a large part of it is delegated to DuckDB, which scans the Parquet files and performs aggregations.
- Iceberg is structured as metadata files, manifest lists, manifests, and data files in a data layer over object storage.
More detail
The slide walks the Iceberg on-disk hierarchy from metadata files down through manifest lists and manifests to the Parquet data files, all reachable from the Postgres instance.
04 / Architecture
- Postgres is process-based: each new connection spawns a process, so embedding DuckDB in every process would waste memory and thread pools.
- Instead pg_lake runs a single side car called the PG Duck server that all Postgres processes connect to for query execution.
- Communication currently uses the Postgres protocol (all local); the team may switch to the Quack protocol later.
- The PG Duck server also holds a local file cache, so files just written to Iceberg can be read back from local disk.
More detail
This shared-side-car design keeps the analytical engine centralized while Postgres remains the front door and the home of all Iceberg logic.
05 / Transactions
- All Iceberg logic is re-implemented from scratch in C rather than reusing DuckDB's Iceberg support, chiefly to support transactions.
- At commit time, writing an Iceberg table also updates the catalog table holding the current metadata location, inside the same transaction as the Postgres changes.
- So you can delete from a Postgres table and insert into an Iceberg table in one atomic commit, with no half-applied or duplicated state.
- The Iceberg catalog table lives inside the Postgres instance and points to metadata JSON, manifests, and Parquet data files in S3 object storage.
- External engines like Spark, Java, and pyiceberg can read the same Iceberg data.
More detail
By keeping the Iceberg catalog inside Postgres and updating it within the user transaction, pg_lake gives lakehouse writes ACID guarantees that normally require complex external systems.
06 / Pipelines & catalogs
- Example pipeline: insert into an unindexed Postgres table (100k-200k rows/sec, ~1-2 ms latency, durable), then a cron job every 10 seconds moves the data into Iceberg.
- After ~10 seconds the data is queryable in Iceberg by other systems or accelerated by DuckDB from Postgres, far simpler than typical pipelines.
- Extra tooling like pg_incremental lets data stay in Postgres and be shoveled into Iceberg incrementally.
- Some engines (e.g. Spark) can use Postgres as an Iceberg catalog, but DuckDB cannot, so the team is adding a REST catalog that wraps the database and can also vend credentials.
- PG Lake metadata can be synchronized into (and back from) Polaris metadata via triggers to expose a REST API; a Duck Lake pull request is open on the repo for people to try out.
More detail
The speaker notes the design resembles Duck Lake and invites the community to test the generated code by trying the open Duck Lake PR.
Key Takeaways
- pg_lake makes Apache Iceberg feel like a native Postgres table format, backed by object storage for effectively bottomless capacity.
- Analytical queries against Iceberg tables are delegated to DuckDB, which scans Parquet and handles aggregations.
- A single shared PG Duck server side car runs DuckDB for all Postgres processes, avoiding per-process memory and thread-pool overhead.
- Iceberg was re-implemented from scratch in C so that writes commit atomically alongside Postgres changes via the in-Postgres catalog.
- Cross-format atomic transactions enable simple, reliable pipelines: fast Postgres inserts plus a cron job that moves data into Iceberg every 10 seconds.
- Postgres works as an Iceberg catalog for engines like Spark; a REST catalog is being added for engines such as DuckDB, with sync to/from Polaris via triggers.
- pg_lake is open source (Snowflake-Labs/pg_lake) and has an open Duck Lake pull request for the community to try.