<#633 feat: Add DuckDB plugin> Pull request opened...
# flyte-github
a
#633 feat: Add DuckDB plugin Pull request opened by andreahlert Summary • Add a DuckDB connector plugin for running SQL queries against DuckDB as Flyte tasks • DuckDB is an embedded analytical database (like SQLite for OLAP) that runs locally and synchronously • Follows the same architecture and patterns as the Snowflake plugin Features • In-memory and file-based database support via
DuckDBConfig(database_path=...)
• Parameterized SQL queries with typed inputs • Extension installation and loading (httpfs, json, spatial, etc.) • Query results returned as pandas DataFrames via temporary parquet files • Automatic cleanup of temporary result files on delete Design Unlike Snowflake/BigQuery, DuckDB runs locally with no remote service, so the connector pattern is adapted: • Queries execute synchronously in
create()
(wrapped in
run_in_executor
for async compat) •
get()
always returns SUCCEEDED since queries complete in
create()
•
delete()
cleans up temporary parquet result files • No credentials, polling, or dashboard links needed Test plan • 25 unit tests covering task creation, config serialization, SQL generation, connector create/get/delete, parameterized queries, extensions, and end-to-end flow • All tests pass (
pytest plugins/duckdb/tests/ -v
) • Linting passes (
ruff check plugins/duckdb/
) • Plugin structure matches Snowflake plugin layout flyteorg/flyte-sdk
#633 Add DuckDB plugin Pull request ready for review by andreahlert flyteorg/flyte-sdk