DuckCon #7
Phages & DuckDB: The mighty duo against the giants
V. Grosboillot, H. Vazquez-Cano · DuckCon #7
8
sections
10
slides
01 / Motivation
- Cancer is today's deadliest disease, but by 2050 drug-resistant bacterial infections are projected to become the leading cause of death.
- Antibiotics are steadily losing effectiveness against resistant bacteria, opening a critical gap in treatment.
- The 'Risks' chart contrasts projected antimicrobial resistance (AMR) mortality (10M by 2050) with today's causes of death like cancer, diarrhoeal disease, and road accidents.
More detail
Source cited on slide: aiehta.com AMR mortality projections.
02 / The biology
- Bacteriophages ('phages') are viruses that infect bacteria: the most abundant organisms on Earth and the natural predators of bacteria.
- A phage binds to a bacterium, injects its genomic code, hijacks the host, replicates, and kills the bacterium in the process.
- Phage therapy offers a complement or alternative to conventional antibiotic treatment.
- Before clinical use, phage genomes must be studied carefully to avoid accidentally introducing genes harmful to patients.
More detail
Eugenie Grosboillot works on the isolation, characterization, and propagation of phages.
03 / The problem
- Phage sequences live in messy, semi-structured text files with hand-made annotations that are inconsistent, sometimes incomplete, and vary by pipeline.
- Bioinformatics pipelines are lengthy: a run can fail an hour in because of one mislocalized or malformed file, which is deeply frustrating.
- Portability matters: the same pipeline must run locally on a laptop and on high-performance computing (HPC) environments.
- The slide maps the full landscape of lab, experiment, and data challenges, from reproducibility to file formats to analysis at scale.
04 / The solution
- Synphage analyzes large collections of phage genomic sequences in a single blow.
- Pipeline stages: GenBank files -> Blastn query results -> Blastp query results -> genome identity -> synteny diagram.
- Each stage produces artifacts (genome.gb, blastn.json, blastp.json, identity.df, synteny.svg) with NCBI/NLM as a reference source.
- DuckDB underpins the workflow, extracting data from messy files, computing gene collinearity, and gathering results for visualization.
- The first pipeline version was very slow and unreliable; DuckDB is what made it faster and more robust.
More detail
Synphage is an open-source project: github.com/vestalisvirginis/synphage
05 / DuckDB approach
- Bioinformatics is a fragmented, file-based ecosystem; DuckDB changes the unit of work from a single file to a whole folder / database.
- It replaces slow Python loops ('the antithesis of parallelism') with lateral, semi, and anti joins.
- Schema inference and sampling remove the human in the loop for deeply nested JSONs.
- A single portable binary runs identically on a laptop and on HPC, preserving product sovereignty and reproducibility.
- Macros and column (COLUMNS) expressions bring simplicity and readability to wide data frames.
06 / Code
- A single query reads all BLAST JSON outputs via read_json_auto('*') wildcards: no per-file Python loops needed.
- Nested JSON fields (program, version, reference, search_target, hits) are extracted directly with the -> path operator.
- Algebraic operations compute percentage_of_identity as round(identity/align_len*100, 3) over sequences of any length.
- A custom blast_top_hit macro obfuscates the similarity-search logic to return the best genome match via a LATERAL join.
- Annotated callouts highlight: wildcards, no loops, no human in the loop, large strings, and macros.
07 / Extra wins
- Analysis at your fingertips: GC content is computed by unnesting nucleotides and applying the built-in HISTOGRAM function.
- The histogram bins A/C/G/T counts (e.g. A=63, C=28, G=35, T=51) and renders an ASCII bar chart right in the CLI.
- Scalability from single large phages to whole phage libraries, across files or table formats.
- Portability and reproducibility via a single binary; DuckDB also mitigates the 'support decay' that kills many academic projects.
- Synteny diagrams (sequence-over-sequence collinearity) are built with column expressions and side-by-side join operations.
08 / Wrap-up
- Every minute saved waiting on a workload brings researchers closer to a publication, a discovery, or the next grant.
- Synphage is open source: github.com/vestalisvirginis/synphage.
- Funded and supported by the University of Ljubljana Biotechnical Faculty, the European Research Council, and Drago's Lab.
- Special thanks to DuckLabs for the invitation to share the DuckDB journey.
Key Takeaways
- Antimicrobial resistance is projected to become the leading cause of death by 2050, making phage therapy a critical research frontier.
- Phage genomes must be analyzed safely before therapeutic use, but their data lives in messy, inconsistent, semi-structured files.
- Synphage turns a slow, fragile, file-based pipeline into a fast, reliable phage genome analytics tool powered by DuckDB.
- DuckDB shifts the unit of work from single files to folders/databases, replacing brittle Python loops with set-based joins.
- Wildcard reads, JSON schema inference, macros, and COLUMNS expressions remove human-in-the-loop toil and boost readability.
- A single portable binary delivers reproducibility from laptop to HPC and guards against academic 'support decay'.
- Faster pipelines directly translate into more publications, discoveries, and funding for scientists.