Can a task read/write from/to a database?
# flytekit-java
b
Can a task read/write from/to a database?
b
yep
g
b
I need this feature from
flytekit-java
, and I'm not sure that exists.
g
oops, sorry. this is flytekit-java channel. i didn’t notice. maybe Rafael can share an example
b
I'm assuming as long as the container has access to the database and the right dependencies are in the deployed workflow, it'll "just work".
b
Exactly. Whatever code you put on the task (and it's dependencies) it's ran in the container
b
the use case I'm thinking about is when a task has to process a large set of data, something where the IO cost around passing via triaditional s3-backed channels makes no sense
b
What are the size of the jars when it's all packaged?
b
Could be 100’s of MB’s
b
Same for docker images. The caveat here is that we're not leveraging the docker caching system, but the base docker image (flytekit-java) will be always cached. Depending on the number of nodes just using pure docker can have the same impact. Will probably need to run some numbers again but comparing both were not that far in warm-up time compared to python, sometimes better sometimes worse but we're running 100s of nodes in our k8s cluster.
For the task itself it will be the same IO cost regardless of what you're using. The only impact in our setup is warm-up times (which depending on the number of k8s nodes python will have the same impact).