{"id":52029333,"url":"https://github.com/tegmentum/synphage-ducklink","last_synced_at":"2026-08-01T11:03:05.465Z","repository":{"id":372400306,"uuid":"1307077619","full_name":"tegmentum/synphage-ducklink","owner":"tegmentum","description":"Rebuilding Synphage's comparative-genomics workflow as WebAssembly components DuckDB calls as SQL table functions","archived":false,"fork":false,"pushed_at":"2026-07-21T01:18:04.000Z","size":422,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-21T02:09:33.406Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tegmentum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-20T23:00:42.000Z","updated_at":"2026-07-21T01:18:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tegmentum/synphage-ducklink","commit_stats":null,"previous_names":["tegmentum/synphage-ducklink"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tegmentum/synphage-ducklink","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tegmentum%2Fsynphage-ducklink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tegmentum%2Fsynphage-ducklink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tegmentum%2Fsynphage-ducklink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tegmentum%2Fsynphage-ducklink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tegmentum","download_url":"https://codeload.github.com/tegmentum/synphage-ducklink/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tegmentum%2Fsynphage-ducklink/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36154735,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-01T02:00:05.789Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-08-01T11:03:04.908Z","updated_at":"2026-08-01T11:03:05.441Z","avatar_url":"https://github.com/tegmentum.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# synphage-ducklink\n\nRebuilding [Synphage](https://github.com/vestalisvirginis/synphage)'s comparative-genomics workflow as WebAssembly components that DuckDB — via DuckLink — calls as ordinary SQL table functions.\n\nNot a fork. Not a wrapper. A reimplementation of the *shape*: the same scientific method, exposed as relations instead of a Python + Dagster + BLAST+ pipeline.\n\n## The contrast\n\n| Synphage today                                    | synphage-ducklink                          |\n|---------------------------------------------------|--------------------------------------------|\n| `pip install` + `apt install ncbi-blast+`         | (once merged) `LOAD ducklink FROM community; SELECT ducklink_load('blast');` — see below for today's local-build path |\n| Dagster server, four sequential jobs              | One SQL query                              |\n| BLAST JSON on disk, parsed via nested paths       | Typed hit rows straight from the aligner   |\n| Biopython + Polars + Pandas glue                  | DuckDB relations end to end                |\n| Linux/macOS/Windows setup differences             | Same `.wasm` everywhere (incl. browser)    |\n\nThe scientific outputs — best-hit selection, gene conservation, synteny diagrams — stay recognisably Synphage. What disappears is the machinery around them.\n\n## Status\n\n- **WIT drafted** — `wit/` defines three biological capabilities (`sequence-search`, `genome-format`, `synteny-renderer`) plus a compound world `blast` that also exports DuckLink's dispatch surface. DuckLink's own WIT is vendored under `wit/deps/duckdb-extension/`.\n- **Three components built,** all targeting `wasm32-wasip2`:\n  - **`blast`** (~319 KB) — exports both `tegmentum:bio/sequence-search` AND `duckdb:extension/{guest,callback-dispatch,table-stream-dispatch}`. Registers `blastn` and `blastp` DuckLink table functions.\n  - **`genome-format`** (~106 KB) — handwritten GenBank parser, exports `tegmentum:bio/genome-format` only (no DuckLink surface yet).\n  - **`synteny-renderer`** (~105 KB) — handwritten SVG generator, exports `tegmentum:bio/synteny-renderer` only.\n- **Filter pushdown live in blast** — `evalue \u003c X`, `percent_identity \u003e= Y`, `query_key IN (…)`, `subject_key IN (…)`, `strand = 'plus'/'minus'` all recognised and folded into `SearchOptions` / early batch pruning / short-circuit. Contradictory clauses (empty key intersection, plus AND minus) trigger a zero-row cursor.\n- **Standalone smoke test via `hosts/blast-test`** — a wasmtime-based Rust binary loads `blast.wasm`, mocks DuckLink's `table_stream` host import, and drives the full protocol (`guest.load` → `call_table_open_filtered` → `call_table_next` → `call_table_close`). Useful for iterating without the DuckLink base build.\n- **Real end-to-end under DuckLink verified.** With `ducklink_{core,cli,loader}.wasm` built and the native `ducklink` host binary, `LOAD blast; SELECT … FROM blastn('[…]','[…]', NULL) WHERE evalue \u003c 1e-6;` returns hits from a real DuckDB engine via wasm. Projection and filter pushdown both observed in the runtime logs (`projection=[6,12,0,1,9,10] nfilt=2`).\n- **Real phage genomes, real biology.** Running BLASTN on 24 CDS from three NCBI phage references (lambda `NC_001416`, T7 `NC_001604`, P22 `NC_002371`) through real DuckLink produces 40 hits with the expected phylogeny: lambda↔P22 shares 6 conserved genes each way (Nin/cIII cassette), T7 finds nothing in either (correct — T7 is a distant podovirus). Full pipeline runs in ~1 s.\n- **Three DuckLink-loadable components in the tree, all fully working end-to-end.**\n  - `blast.blastn/blastp` — 40 hits across 24 CDS from three real NCBI phage genomes.\n  - `synteny-renderer.render_synteny_svg` — produces 610 B of SVG on a tiny test model (`bytes_len=610`, `svg_len=610`); empty inputs correctly return zero rows.\n  - `genome-format.genbank_scan(contents VARCHAR)` — parses all three phage genomes: NC_001416 → 285 features / 73 CDS, NC_001604 → 162 / 60, NC_002371 → 191 / 72.\n- **Repeatable real-DuckLink acceptance test.** `acceptance/run-ducklink.sh` drives the full pipeline (24 CDS from lambda / T7 / P22 → `blastn` under real DuckLink → conservation views → assertions) in a single DuckLink invocation and prints `ACCEPTANCE-DUCKLINK PASSED` on success. The mock-host driver `acceptance/run.sh` remains for fast iteration when the DuckLink base isn't built.\n- **Pure-SQL acceptance test (no Python).** `acceptance/run-sql-only.sh` runs the whole pipeline — read → parse → BLAST → conservation → summary → assertions — as **one DuckLink SQL script** (`acceptance/pipeline-sql-only.sql`). Uses `SET VARIABLE x = (SELECT content FROM read_text(...))` + `getvariable('x')` to pass computed values through DuckDB's TVF-argument binder (which otherwise rejects subqueries). Produces the same 40-hits/8-conserved result. This is the closest DuckDB currently permits to \"the whole scientific pipeline is one query\"; the remaining plural-statement shape is a DuckDB binder rule, not intrinsic to the design.\n- **Native DuckDB variant.** `acceptance/run-native-duckdb.sh` runs the same pure-SQL pipeline through the native `duckdb` CLI + the `ducklink.duckdb_extension` native community extension (no DuckLink wasm CLI). This is the daily-driver shape once ducklink lands in `duckdb/community-extensions`. Uses DuckDB's built-in `to_json(list(struct_pack(...)))` for the queries payload — no `jsonfns` extension needed. Same 40 hits, same conservation, `ACCEPTANCE-NATIVE-DUCKDB PASSED`.\n- **Thin SQL sugar** at `sql/blast_macros.sql` (`blastn_of`, `blastp_of`). Table macros over the three JSON string args, with default `opts_json := NULL`. Fuller sugar (`blastn(TABLE genes)`) is blocked by DuckDB's binder rejecting subqueries in TVF-arg positions — the moment that lifts, the natural signature drops in without extension-side changes.\n- **Two DuckLink-shape workarounds now understood and documented,** both applied uniformly across the three components:\n  - **`Logicaltype::Complex(\"LIST(STRUCT(…))\")` flattens to `VARCHAR[]`** at the DuckDB binder in DuckLink 4.0.0 — the STRUCT payload is lost. All three components now take VARCHAR JSON args instead; the `Duckvalue::Complex` code path is still there for when DuckLink learns to preserve richer type-expressions.\n  - **`std::fs::read` inside a component can't reach files** — WASI preopens (`ducklink --dir HOST::GUEST`) don't thread through to extension instances, and DuckLink's `files` interface is for replacement-scan registration only, not general file reads. Idiomatic fix: extension takes file *content*, not path. `genome-format.genbank_scan(contents VARCHAR)` accepts raw GenBank text; users would compose with DuckDB's native `read_text()` if TVF-arg subqueries were allowed (currently a separate DuckDB binder limitation — literal inlining works today, subquery-composition becomes daily-driver the moment that binder rule lifts).\n- **Synphage conservation SQL ported** — `sql/best_hits.sql`, `sql/gene_conservation.sql`, `sql/summary.sql`. Run against DuckDB 1.5.4 fixture data cleanly.\n\n### DuckLink caveat: complex arg types\n\nDuckLink 4.0.0's `Logicaltype::Complex(\"LIST(STRUCT(...))\")` currently flattens to `VARCHAR[]` at the DuckDB binder — the STRUCT payload is lost. Our first cut used the natural LIST(STRUCT) shape and hit a type-signature mismatch. Working around it: `blastn` now takes three VARCHAR args carrying JSON payloads. Consumers write:\n\n```sql\nLOAD blast;\nSELECT * FROM blastn(\n    '[{\"key\":\"q1\",\"data\":\"ACGT...\"}]',\n    '[{\"key\":\"s1\",\"data\":\"ACGT...\"}]',\n    NULL\n);\n```\n\nA DuckDB SQL macro that hides the `to_json()` wrapping is the natural next sugar, and the parser still accepts `Duckvalue::Complex` so this switch is reversible once DuckLink learns to preserve arbitrary type-expressions.\n\nSee [`DESIGN.md`](DESIGN.md) for the rationale behind the WIT shapes and the target demo query.\n\n## Layout\n\n```\nsynphage-ducklink/\n├── README.md, DESIGN.md\n├── wit/\n│   ├── world.wit             # 4 biological-only worlds + compound `blast` world\n│   ├── sequence-search.wit   # BLASTN + BLASTP (implemented)\n│   ├── genome-format.wit     # GenBank -\u003e four relations (implemented, biology-only)\n│   ├── synteny-renderer.wit  # tracks/features/links -\u003e SVG (implemented, biology-only)\n│   └── deps/duckdb-extension/  # vendored DuckLink WIT (52 files)\n├── components/\n│   ├── blast/                # sequence-search + DuckLink dispatch\n│   │   └── src/{lib,align,scoring,strand,filter,ducklink,pushdown,bindings}.rs\n│   ├── genome-format/        # handwritten GenBank parser (biology-only)\n│   │   └── src/{lib,parser,location,model,bindings}.rs\n│   └── synteny-renderer/     # handwritten SVG renderer (biology-only)\n│       └── src/{lib,render,bindings}.rs\n├── hosts/\n│   └── blast-test/           # standalone wasmtime host for end-to-end validation\n├── examples/\n│   └── tiny-blastn/          # small sample FASTA files for the test-host\n└── sql/\n    ├── best_hits.sql         # per-(query, subject-genome) ranking view\n    ├── gene_conservation.sql # (query gene × subject genome) grid + LEFT JOIN back to features\n    ├── summary.sql           # per-gene rollup: n_other_genomes, conservation_pct, avg identity\n    └── README.md             # what each file computes + assumed schemas\n```\n\n## Roadmap\n\n1. ✅ **BLASTN + BLASTP wasm component** targeting `sequence-search-only`.\n2. ✅ **DuckLink table-function registration** — `blastn(...)`, `blastp(...)` at the SQL surface.\n3. ✅ **Filter pushdown** — evalue ceiling, identity floor, key restrictions, strand, with short-circuit.\n4. ✅ **Standalone end-to-end smoke test** via `hosts/blast-test` (validates dispatch surface without DuckLink base build).\n5. ✅ **Conservation SQL** ported into `sql/` — best-hit ranking, gene-conservation grid, per-gene summary.\n6. ✅ **`genome-format` wasm component** — handwritten GenBank parser exporting biology-only.\n7. ✅ **`synteny-renderer` wasm component** — handwritten SVG generator exporting biology-only.\n8. **DuckLink base build + real E2E** — build `ducklink_{cli,core,loader}.wasm` in `../ducklink/`, then run `SELECT * FROM blastn(…)` under real DuckLink. Requires the sibling repo's substantial build.\n9. **DuckLink bridges for `genome-format` and `synteny-renderer`** — add the compound world + dispatch impls the same way `blast` got them, so `SELECT * FROM genbank_scan('*.gb')` and `SELECT render_svg(…)` work.\n10. **Acceptance test** — run the demo query in `DESIGN.md` on real Synphage example genomes; compare outputs to the reference Synphage pipeline.\n\n## Related\n\n- [`synphage`](https://github.com/vestalisvirginis/synphage) — the workflow this reimplements.\n- [`scry-webfunctions-demo`](../scry-webfunctions-demo) — the rust-bio BLASTP wasm component this project extends.\n- [`webfunction-wit`](../webfunction-wit), [`stardog-webfunction-wit`](../stardog-webfunction-wit) — the SPARQL-shaped WIT the BLASTP component currently targets. The `tegmentum:bio` package in this repo is the DuckLink-shaped sibling.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftegmentum%2Fsynphage-ducklink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftegmentum%2Fsynphage-ducklink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftegmentum%2Fsynphage-ducklink/lists"}