I am evaluating Flyte for data engineering. When i...
# flyte-supportq
quick-byte-5908
07/01/2023, 2:56 PMI am evaluating Flyte for data engineering. When is Flyte not a good choice (compared to Dagster - another candidate)? The data science group uses Metaflow and they are actively advocating for us to adopt it too. Is Flyte comparable to data science tools like Metaflow apart from the common workflow orchestration bit?
f
freezing-airport-6809
07/02/2023, 6:33 AM
It is absolutely comparable to metaflow and is probably better at ML related tasks than metaflow. It’s definitely more on thr ML side than data engineering
freezing-airport-6809
07/02/2023, 6:41 AM
For example take a look at pytorch, onnx, tensorflow integrations and also data frames. Also map tasks, spark etc. it has opinionated docker build system for folks who want to use it, it maintains strong reproducibility and is fast to Iterate on.
We would love to know from the data science team if they think anything in Flyte can be improved for them.
freezing-airport-6809
07/02/2023, 6:42 AM
I think the community would be happy to chime in here
q
quick-byte-5908
07/04/2023, 5:05 PMAre you able to share more specifics? I can't go back to my team with high-level generalizations like "absolutely comparable and probably better", that would look quite stupid on my part 🙂 They were on Kubeflow before which had these integrations too, but they never worked well in practice.
quick-byte-5908
07/04/2023, 5:06 PMTo the original question - is it not recommended to use Flyte for ETL use cases then?
f
freezing-airport-6809
07/05/2023, 11:42 AM
Sadly kubeflow made it bad for everyone
a
average-finland-92144
07/06/2023, 4:42 PM@quick-byte-5908 This is an example implementation of ETL using Flyte https://docs.flyte.org/projects/cookbook/en/latest/getting_started/data_engineering.html#data-engineering that could probably give you an idea.
642 Views