and another question, if i somehow manage to make work perfectly fine, what is the benefit i am getting from that?
01/23/2022, 4:48 PM
This is a great question. So for this let's try and understand what we are doing and then the why
What and why (orthogonal to Flyte)
When you define functions you are creating reusable and self dependent snippets that are easy to test in isolation and probably can be reused. Another thing is they are easier to read and follow and even document. This is why you should usually break down your monolith script into functions.
What and why with Flyte.
So let's say you have a pipeline - when you take some text data apply some transformations, train a model or use an existing model to perform predictions.
You may want to break down your monolith pipeline into these logical
Every function need not be a task.
Now what you get is failure isolation, recoverability and automated documentation about what your pipeline does in a logical sense
Tasks offer automated checkpoints and so in case of failures
Only the task will be rerun. Flyte will also analyze the data dependency and then run tasks in parallel.
Also you may want a larger computer for some processing - and smaller for other tasks, you can annotate the resources on your tasks
And the final thing, with Flyte everything is portable to the cloud. So now you can use the Flyte Ui to trigger an execution anytime, observe inputs and outputs to every task and have many concurrent executions that are guaranteed to run without your laptop even on
Cc @Samhita Alla
01/24/2022, 9:44 AM
this seem to be very helpful and kinda makes sense now for 'why am i using'