Hello We have a workflow that fails with ```Timeout in node` Flyte #ask-the-community

Hello! We have a workflow that fails with ```Timeo...

Mücahit

07/18/2023, 1:22 PM

Hello! We have a workflow that fails with

Copy code

Timeout in node

but the task that fails gets marked as Unknown, so we can't click to it and refer to the logs etc. and Flyte(1.8) keeps tracking the task

Mücahit

07/18/2023, 1:23 PM

task that fails has a 1h timeout but it does not marked as failed or timed out or sth like that and the duration that shows 11h now keeps getting increased(another execution is like 420h now)

Mücahit

07/18/2023, 1:23 PM

I would expect the task to be marked as failed but it seems like Flyte looses track of the task

Samhita Alla

07/19/2023, 5:08 AM

Is it possible for you to check the propeller logs?

Mücahit

07/19/2023, 7:39 AM

image.png

Mücahit

07/19/2023, 7:39 AM

@Samhita Alla these logs look quite relavant

Mücahit

07/19/2023, 7:43 AM

and these

Samhita Alla

07/19/2023, 8:25 AM

Can you set

inject-finalizers

to true in the flyte propeller config? More info available in the attached thread: https://discuss.flyte.org/t/2743027/hi-we-re-doing-some-performance-testing-and-when-we-start-a-#2409e7df-fda2-475e-a184-49e1be51e3ff

Mücahit

07/19/2023, 9:02 AM

just did it, I'll see if will fix the issue tomorrow

Mücahit

07/19/2023, 9:03 AM

but also, task is a spark task, will this still work 🤔

Samhita Alla

07/19/2023, 9:18 AM

Are you asking if timeout works for spark task?

Mücahit

07/19/2023, 9:20 AM

If adding

inject-finalizers

will make Flyte to mark the spark task as failed instead of it getting stuck at

unknown

Samhita Alla

07/19/2023, 11:45 AM

I'm not sure; let's check if it's working for you. If not, we can think about how to resolve the issue.

Mücahit

07/20/2023, 6:08 AM

@Samhita Alla this seems to have fixed the issue! Thank you very much! I am also thinking maybe

inject-finalizers

should be enabled by default

Samhita Alla

07/20/2023, 6:19 AM

@Dan Rammer (hamersaw), what do you think?

Mücahit

07/20/2023, 10:18 AM

It was a bit too early to say that it worked, now I see the same, state in the UI with these log ``````

Samhita Alla

07/21/2023, 4:07 AM

Could you share the updated plugin config, please?

Mücahit

07/21/2023, 12:43 PM

sure, this is the flyte-propeller-config configmap's relevant section

Copy code

k8s.yaml: |
    plugins:
      k8s:
        inject-finalizers: true

Samhita Alla

07/21/2023, 12:52 PM

I think it needs to be

inject-finalizer

but not

inject-finalizers

Dan Rammer (hamersaw)

07/24/2023, 1:48 PM

@Mücahit I'm having some difficulty following here. It seems all of the propeller logs you have linked are for different workflows? Some are for

python-task

, some for

athena

task, but you have mentioned spark tasks - are there issues with all of these? Is the timeout not working for everything?

Mücahit

07/26/2023, 10:17 AM

@Samhita Alla true.. I changed the config but the timeout issue hasn't ocurred yet, it might happens once more this week

Mücahit

07/26/2023, 10:19 AM

@Dan Rammer (hamersaw) it is one workflow with multiple tasks,

spark

and

python

as you can see in the screenshot. Workflow gets marked as failed with Timeout in node error message but the task that times out

read_data

one(spark-task) is not marked as failed but stays as

UNKNOWN

infinitely .

Mücahit

07/27/2023, 6:47 AM

and the issue happened again, with inject-finalizer configuration

Dan Rammer (hamersaw)

07/27/2023, 10:38 PM

@Mücahit so rereading this - there is a task that times out, the workflow fails with an error saying "Timeout in node", and the concern is that there is a task running in the background that Flyte lost track of?

Dan Rammer (hamersaw)

07/27/2023, 10:39 PM

Does the Spark CR get created in the k8s cluster? Can you verify it exists and is still around when the workflow fails?

Dan Rammer (hamersaw)

07/27/2023, 10:41 PM

I suspect there is never a k8s resource created because otherwise the task status would be something like

QUEUED

RUNNING

, the

UNKNOWN

leads me to believe that the task never began executing. and the increasing timestamp in the UI does not accurately reflect what is actually happening. If Flyte reports that a node started but doesn't have a node ending this duration will continue to tick in the UI. There may be a bug where on timeout Flyte misses a report for the task ending, but there is actually nothing executing.

Lee Ning Jie Leon

08/03/2023, 5:50 AM

I hit into a similar issue. Our downstream had issue and were running slowly, so all the task (ShellTask) took longer time than usual. After the downstream was fixed, rerunning the job works. The UI timer is increasing infinitely and seems to be a visual bug only. We do not have the inject-finalizer config. I believe it went from

Running->TimingOut->TimedOut->Unknown

From propellor logs, attached a screenshot. After which it triggers deletion and marked the workflow as failed.

Copy code

Change in node state detected from [Running] -> [NodePhaseTimingOut], (handler phase [Timedout])
Recording NodeEvent [node_id:"n1" execution_id:<project:"..." domain:"production" name:"f4fb0fb246163cd71000" > ] phase[UNDEFINED]

Concern • Zero logs of what is happening being propagated to UI or why

Timeout in node

is happening • Metrics doesn't show in

flyte:propeller:all:node:failure_duration_ms

that executions failed at all. Metrics shows all execution for this workflow to be all successful. It seem that

inject-finalizer

flag might resolve the issue above, is there a reason why this is not enabled by default? Want to understand if there are any downside to it before i switch it on.

34 Views

Open in Slack

Previous Next