https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
l

Louis DiNatale

12/06/2022, 5:53 PM
I noticed in the flyte UI that dynamics are grouped in “attempts” what does this mean?
d

Dan Rammer (hamersaw)

12/06/2022, 6:54 PM
Can you provide a little more context? I'm not sure I fully understand.
l

Louis DiNatale

12/06/2022, 6:56 PM
message has been deleted
When running a dynamic at a very large scale we see these “attempts”
y

Yee

12/06/2022, 8:39 PM
do the earlier attempts fail?
l

Louis DiNatale

12/06/2022, 8:40 PM
A few
not the entire run just a few long running jobs from a timeout we need to fix but a few attempts had no failures and another attempt would happen after
y

Yee

12/06/2022, 8:41 PM
dan - retries for dynamic, do they re-compute the spec?
d

Dan Rammer (hamersaw)

12/07/2022, 5:00 PM
So what seems to be happening here is that you have some dynamic task (ex. 
n0
) that generates a number of subtasks (ex. 
n0-0
, 
n0-1
, and 
n0-2
). If when executing the dynamic task subtasks 
n0-0
and 
n0-1
succeed but 
n0-2
fail then the top level dynamic task (ie. 
n0
) fails. This is then retried, which causes all of the subtasks to be executed again (even those that previously succeeded), hence multiple attempts for each subtask. Does this sound correct? I think it would make sense to not retry the top-level dynamic task after the subtask workflow closure has been generated. If one of the subtasks fail, then the dynamic task should fail. Is this what you would expect? I know this is an abstraction that we use in map tasks.
@Yee, yes it does look like the spec is recomputed. This is obviously an inefficiency, but it should not break correctness because none of the subtask executions are recovered.
l

Louis DiNatale

12/07/2022, 6:12 PM
This makes a lot of sense, so even if one retry happens it will cascade to multiple jobs.
d

Dan Rammer (hamersaw)

12/07/2022, 6:56 PM
Louis, would you mind filing an issue for this? I certainly think it is functionality that should be updated. Since it doesn't effect correctness it probably won't happen in the next few days, but it would be nice to have on the roadmap 😄.
l

Louis DiNatale

12/07/2022, 6:56 PM
Ya no problem!
Ill do it later today.
Powered by