full-evening-87657
10/16/2023, 9:13 AMcalm-pilot-2010
10/16/2023, 10:17 PMSQLAlchemyTask
? Do you want to write the output of a task to Postgres or do you want a task which just runs a Postgres query? My understanding of the SQLite3Task
and SQLAlchemyTask
is that they just run a query against the database.
If you want to store the output of a python task in postgres then I think it would be possible to make yourself a StructuredDataset
output format similar to the one for bigquery https://github.com/flyteorg/flytekit/blob/d90385a0e921a6a373ab80a9261949d45fbbad92/tests/flytekit/unit/types/structured_dataset/test_bigquery.py#L17-L18.
I'm expecting to do something like this in future and I think the most likely solution for me is writing a decorator completely distinct from flyte so that I can replicate StructuredDataset
output types to postgres in addition to writing them to an object store.