#2865 [Core feature] Add support for duckdb with structured dataset Issue created by pingsutw Motivation: Why do you think this is important? duckdb is a serverless DB deployed at GCP. we could integrate it with Flyte, and write task output, like

pandas.DataFrame

or

arrow.Tables

, to duckdb transparently.

@task
def t1() -> Annotated[StructuredDataset, kwtypes(len=int)]:
    df = pd.DataFrame({"name": ["dylan", "steve"],"age": [33, 32]})
    return StructuredDataset(df, uri=duckdb_uri) # flytekit will write pandas dataframe to duckdb

Goal: What should the final outcome look like, ideally? Add a duckDB plugin in flytekit, and add a structured dataset encoder and decoder. Here is an example to add custom encoder/decoder in flytekit-plugin Describe alternatives you've considered No response Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte