I m trying to debug a task failure it stopped after 1h15min Flyte #ask-the-community

I'm trying to debug a task failure: it stopped aft...

Felix Ruess

01/05/2023, 8:39 PM

I'm trying to debug a task failure: it stopped after 1h15min and I can't seem to find why. Is there a timeout set in flyte by default somewhere?

Dan Rammer (hamersaw)

01/05/2023, 8:42 PM

there are node execution deadlines but they default to something like 24h. so this would not be it. what kind of task is it?

Felix Ruess

01/05/2023, 8:43 PM

A ContainerTask. In the flyte console it just tells me: Some node execution failed, auto-abort.

Felix Ruess

01/05/2023, 8:44 PM

Ah, wait... does flyte abort all tasks in a workflow if another parallel one fails?

Dan Rammer (hamersaw)

01/05/2023, 8:44 PM

ok, do you know which node in the workflow failed? if one fails flyte aborts all running nodes in the workflow. there is a configuration option to allow the workflow to make as much progress as possible (ie. not abort currently running nodes)

Dan Rammer (hamersaw)

01/05/2023, 8:44 PM

^^^ exactly

Dan Rammer (hamersaw)

01/05/2023, 8:45 PM

the configuration is called failure policy and can be set on the workflow - i'll find the configuration quick

Dan Rammer (hamersaw)

01/05/2023, 8:47 PM

here are the available failure policies and it looks like they're set here with

failure_policy

Felix Ruess

01/05/2023, 8:51 PM

ah, thanks @Dan Rammer (hamersaw) Can I change the default somewhere?

Dan Rammer (hamersaw)

01/05/2023, 8:55 PM

So right now I do not believe we expose a default configuration. It should be a relatively easy add if you're looking to make another contribution, otherwise it would be great to file an issue to get this on the roadmap.

Felix Ruess

01/05/2023, 9:08 PM

hm... looks like flyteidl would need to be adapted for that...

Felix Ruess

01/05/2023, 9:09 PM

or were you thinking a default in flytekit somehow?

Felix Ruess

01/05/2023, 9:17 PM

I guess fail_immediately is a good default for most cases, just in my case this container task is not idempotent and I don't want it... IMHO there are bigger fishes to fry and changing flyteidl here is quite a bit more work than my case warrants... I'll just set in on the workflow decorator.

Felix Ruess

01/05/2023, 9:17 PM

Thanks for the quick help though!

Yee

01/05/2023, 10:30 PM

what are the other fish out of curiosity?

Yee

01/05/2023, 10:31 PM

#3065?

Felix Ruess

01/05/2023, 10:41 PM

That as well, but to me more importantly the possibility to add a pod spec to ContainerTasks or otherwise specify runetimeClass and affinity... That is a blocker right now

Yee

01/06/2023, 1:49 AM

do pod templates not work for that?

Felix Ruess

01/06/2023, 10:55 AM

For the runtimeClass, I use pod templates as a workaround. But affinity I need to set per task.

177 Views

Open in Slack

Previous Next