DuckCon #7
Native app development using DuckDB
Milla Henriksson · DuckCon #7
8
sections
11
slides
01 / Intro
- Speaker builds software at Toyota Gazoo Racing, the WRC team.
- The project is a native desktop app for rally-car engineers, built around DuckDB.
- Talk gives a native-app-development perspective on how DuckDB fits this use case.
02 / Requirements
- Needed a hybrid offline/online tool with an embedded, OLAP-style database.
- Also had to try Microsoft Fabric, so connectivity to Fabric was required.
- Security mandate: access had to be strictly user-based.
- Final stack: Tauri + DuckDB + Microsoft Fabric (Delta), with DuckDB embedded in Tauri's Rust backend.
- Data storage uses Lakehouses inside OneLake with a Delta backend, reached over the ABFSS scheme.
More detail
Tauri is a framework for native apps with a Rust-based backend; DuckDB runs embedded in Tauri's Rust process via the Rust client API.
03 / Concurrency
- Tauri is multi-process; the first approach used application-level locking and async lock acquisition.
- To honor DuckDB's model, the DB now runs on a single dedicated thread.
- Other threads send jobs to the worker via a Rust MPSC sender, passed as heap-allocated closures.
- The worker thread executes those closures against the single database connection.
- Local DB files are attached with encryption (AES GCM cipher) and tables are created; offline the system is ready.
More detail
start_db_thread spawns a worker looping over an MPSC receiver; init_db runs ATTACH ... ENCRYPTION_KEY / ENCRYPTION_CIPHER 'GCM' and CREATE TABLE IF NOT EXISTS.
04 / Auth
- Authorization follows the RFC for OAuth 2.0 for native apps, including the PKCE extension.
- OpenID Connect is layered on top for authentication.
- The flow (DuckDB, Data Tool, Browser -> authorization/token endpoints) yields a per-user access token.
- That access token is set inside DuckDB, satisfying the user-derived access requirement for Fabric.
05 / Upload
- Engineers' data lives in files (e.g. Excel); the tool ingests these files as input.
- The reqwest crate uploads the raw files over HTTPS to the OneLake Lakehouse /Files area.
- DuckDB then appends the data to Lakehouse /Tables using URLs with the Azure blob (ABFSS) scheme.
- This leverages the Delta and Azure extensions.
06 / Insert
- CREATE OR REPLACE SECRET sets an Azure secret with the per-user access token and account name 'onelake'.
- Data is written into a temporary staging table via an appender.
- ATTACH IF NOT EXISTS binds the remote ABFSS Delta table, then INSERT INTO remote SELECT * FROM the staged table.
- The remote table is detached once the append completes.
More detail
Uses the Delta and Azure extensions; the remote is attached as TYPE delta over an 'abfss://.../Tables/measurement' URL.
07 / Sync back
- To ingest from remote Delta tables, the delta_scan function is used inside a SELECT query.
- delta_scan reads a Lakehouse Delta table like any other table, using the ABFSS URL.
- Rows are fetched and collected into typed frames, then the local tables are rebuilt.
- The same Azure secret (per-user token) authorizes the read.
08 / Takeaways
- Considerations: a couple of hard-to-trace issues surfaced, but docs are good and help is a GitHub ticket away.
- Successes: embedded OLAP, encryption, and Fabric support with user-derived access.
- All original requirements were met; DuckDB fits the native-app tech stack very well.
Key Takeaways
- DuckDB works well as an embedded OLAP engine inside a Tauri (Rust) native desktop app.
- Running DuckDB on a single worker thread fed by an MPSC channel honors its concurrency model better than app-level locking.
- Local database files can be attached with encryption using an AES GCM cipher.
- OAuth 2.0 for native apps with PKCE plus OpenID Connect delivers the required per-user access token, which is set directly in DuckDB.
- The Delta and Azure extensions let DuckDB append to and scan Microsoft Fabric OneLake Delta tables over the ABFSS scheme.
- Raw file upload uses the reqwest crate over HTTPS, while table-level sync uses ATTACH/INSERT and delta_scan.
- Overall DuckDB satisfied all requirements: embedded OLAP, encryption, and user-derived Fabric access.