DuckCon #7
Effortlessly modernize your legacy
Nicolas Renkamp · DuckCon #7
7
sections
8
slides
01 / The Enterprise Pirate
Bringing DuckDB into a large enterprise
- Nicolas Renkamp leads Data & AI Platforms at Merck (Darmstadt), and calls himself an 'enterprise pirate' who wants DuckDB inside a big company, not just in startups.
- Merck spans three sectors: life science, health care, and electronics, and all three now run DuckDB feeding critical production applications.
- The talk shares Merck's journey adopting DuckDB in existing pipelines and how SQLFrame made the transition fast.
- Motivation: single-node compute performance gains and DuckDB blog posts made the case too strong to ignore.
02 / Do We Even Need Distribution?
The legacy platform and the single-node question
- In 2025 Merck's data lake hit the 10-year mark; PySpark carried the core transformation logic feeding hundreds of data products and thousands of use cases.
- A diverse engineering skill set means technology choices persist; you cannot swap frameworks in a day.
- Doing the math: median query scans ~55 MB and the 99.9th percentile scans ~35 GB, well below the thresholds where a distributed engine is needed.
- Referencing Hannes Muhleisen's 'The Lost Decade of Small Data?', the conclusion is that most workloads fit comfortably on a single node.
More detail
The two challenges to solve: (1) run DuckDB instead of Apache Spark, and (2) convert the existing PySpark code to SQL that DuckDB can execute.
03 / Narrowing the Attack Surface
Simplified infra and two focus areas
- Moving to DuckDB simplifies the stack: no clusters, just the right EC2 instances with NVMe SSDs plus spill-to-disk configuration.
- Rather than migrating everything, they narrowed the scope to two focus areas based on proof-of-concept results.
- Focus 1: highly standardized transformations run thousands of times daily on tables like SAP data (e.g. trimming strings, converting bad values to real NULLs).
- Focus 2: expensive jobs that 'shouldn't be', where Spark was over-shuffling, spilling terabytes to disk, and missing business SLAs.
More detail
The metrics panel showed one job class with ~70 days aggregate runtime, 8h duration, 5.7 TB disk spillage and 5.1 TB shuffle write across all tasks, despite modest input and output sizes.
04 / SQLFrame + SQLGlot
Translating PySpark to DuckDB SQL
- SQLFrame implements the full PySpark DataFrame API but emits DuckDB SQL, with no dependency on Spark clusters or even the Spark library.
- It builds on SQLGlot, a battle-tested open-source SQL parser, transpiler and optimizer created by Toby Mao (co-founder of SQLMesh/Tobiko).
- SQLGlot can, for example, take a Snowflake-dialect query and transpile it into equivalent DuckDB SQL.
- SQLFrame is created and maintained by Ryan Eakman; the flow is PySpark code -> SQLFrame -> SQLGlot -> DuckDB SQL.
05 / The Blueprint
Separation of concerns between platform and data teams
- The platform team owns setup: temp directory on SSD, performance settings (e.g. disabling insertion-order preservation), and S3 read access.
- The data engineering team keeps writing essentially the same PySpark code as before.
- The platform layer takes the generated SQL and pipes it into a COPY statement that DuckDB executes.
- For production robustness, the recommendation is to use DuckLake or Iceberg to gain their built-in fail-safes.
06 / Production Impact
Measured results on real workloads
- SAP standardized transformations dropped from ~8 million weekly compute-seconds to roughly 2 million: about 75% less compute, while running more jobs (600 vs 500).
- DuckDB wins because it vectorizes queries better, carries no distribution overhead, and packs 10+ years of database research.
- One expensive job unioning 18 datasets with left joins produced ~11,000 lines of generated SQL.
- That job went from ~7 hours to consistently under 1 hour, using far fewer combined CPU and memory resources.
07 / Take the DuckDB Pill
Why it worked and what to take home
- The 'dopamine hit' of feeding in 9-year-old legacy code, getting SQL out, and running it 10x faster is what kept the migration going.
- Success came from focusing on specific workloads and leaving surrounding infrastructure boxes largely intact.
- This is a first step toward a more sovereign data stack with DuckDB as the engine, with plans to explore DuckLake next.
- Closing invitation: fellow 'enterprise pirates' should join the ship and take the DuckDB pill.
Key Takeaways
- Most enterprise analytics workloads fit on a single node: Merck's median query scanned ~55 MB and its 99.9th percentile only ~35 GB.
- You do not have to rewrite legacy PySpark by hand: SQLFrame reproduces the PySpark DataFrame API and emits DuckDB SQL via SQLGlot.
- Scope the migration; targeting standardized transformations and pathologically expensive Spark jobs yields fast, high-confidence wins.
- A platform-vs-data-team separation of concerns lets engineers keep their code while the platform owns DuckDB config and S3/COPY plumbing.
- Real gains at Merck: ~75% less weekly compute on SAP transforms and a 7h -> sub-1h drop on a heavy multi-dataset union job.
- Moving to DuckDB simplifies the stack: no clusters, just NVMe-backed EC2 instances with spill-to-disk.
- For production durability, layer DuckLake or Iceberg on top to inherit their fail-safes.