I am evaluating Flyte for data engineering. When i...
# ask-the-communityf
foo bar
07/01/2023, 2:56 PMI am evaluating Flyte for data engineering. When is Flyte not a good choice (compared to Dagster - another candidate)? The data science group uses Metaflow and they are actively advocating for us to adopt it too. Is Flyte comparable to data science tools like Metaflow apart from the common workflow orchestration bit?
k
Ketan (kumare3)
07/02/2023, 6:33 AM
It is absolutely comparable to metaflow and is probably better at ML related tasks than metaflow. It’s definitely more on thr ML side than data engineering
Ketan (kumare3)
07/02/2023, 6:41 AM
For example take a look at pytorch, onnx, tensorflow integrations and also data frames. Also map tasks, spark etc. it has opinionated docker build system for folks who want to use it, it maintains strong reproducibility and is fast to Iterate on.
We would love to know from the data science team if they think anything in Flyte can be improved for them.
Ketan (kumare3)
07/02/2023, 6:42 AM
I think the community would be happy to chime in here
f
foo bar
07/04/2023, 5:05 PMAre you able to share more specifics? I can't go back to my team with high-level generalizations like "absolutely comparable and probably better", that would look quite stupid on my part 🙂 They were on Kubeflow before which had these integrations too, but they never worked well in practice.
foo bar
07/04/2023, 5:06 PMTo the original question - is it not recommended to use Flyte for ETL use cases then?
k
Ketan (kumare3)
07/05/2023, 11:42 AM
Sadly kubeflow made it bad for everyone
d
David Espejo (he/him)
07/06/2023, 4:42 PM@foo bar This is an example implementation of ETL using Flyte https://docs.flyte.org/projects/cookbook/en/latest/getting_started/data_engineering.html#data-engineering that could probably give you an idea.
468 Views