Can a task read/write from/to a database?
# flytekit-javab
bumpy-match-83743
08/14/2024, 2:59 PMCan a task read/write from/to a database?
b
brief-window-55364
08/14/2024, 3:12 PMyep
g
glamorous-carpet-83516
08/14/2024, 5:38 PMcheck out this sql task. https://docs.flyte.org/en/latest/flytesnacks/examples/sql_plugin/sql_alchemy.html#sql-alchemy
b
bumpy-match-83743
08/14/2024, 5:39 PMI need this feature from
flytekit-javag
glamorous-carpet-83516
08/14/2024, 5:42 PMoops, sorry. this is flytekit-java channel. i didn’t notice. maybe Rafael can share an example
b
bumpy-match-83743
08/14/2024, 8:16 PMI'm assuming as long as the container has access to the database and the right dependencies are in the deployed workflow, it'll "just work".
b
brief-window-55364
08/14/2024, 8:46 PMExactly. Whatever code you put on the task (and it's dependencies) it's ran in the container
b
bumpy-match-83743
08/14/2024, 8:48 PMthe use case I'm thinking about is when a task has to process a large set of data, something where the IO cost around passing via triaditional s3-backed channels makes no sense
b
brief-window-55364
08/14/2024, 10:57 PMWhat are the size of the jars when it's all packaged?
b
bumpy-match-83743
08/15/2024, 3:08 AMCould be 100’s of MB’s
b
brief-window-55364
08/15/2024, 9:15 AMSame for docker images. The caveat here is that we're not leveraging the docker caching system, but the base docker image (flytekit-java) will be always cached.
Depending on the number of nodes just using pure docker can have the same impact. Will probably need to run some numbers again but comparing both were not that far in warm-up time compared to python, sometimes better sometimes worse but we're running 100s of nodes in our k8s cluster.
brief-window-55364
08/15/2024, 9:17 AMFor the task itself it will be the same IO cost regardless of what you're using. The only impact in our setup is warm-up times (which depending on the number of k8s nodes python will have the same impact).