Hi We added a timeout for one node in our workflow This node Flyte #announcements

Hi! We added a timeout for one node in our workflo...

millions-painter-84488

03/29/2022, 10:17 AM

Hi! We added a timeout for one node in our workflow. This node is handled by a backend plugin (

bigqueryjob

). We see in the logs that the node timeouts as expected:

Copy code

"Current execution for the node timed out; timeout configured: 3h0m0s"

However, the

Delete

method of the plugin (https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/webapi/bigquery/plugin.go#L226) is never called after the timeout, so the job is not cancelled. Is this expected? How can I tell propeller to abort the node after a timeout?

millions-painter-84488

03/29/2022, 10:20 AM

We can see in the internal docs that:

Copy code

// Delete the object in the remote service using the resource key. Flyte will call this API at least once. If the
	// resource has already been deleted, the API should not fail.

https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.26/go/tasks/pluginmachinery/webapi#AsyncPlugin So we would’ve expected propeller to call this method after a time out.

hallowed-mouse-14616

03/29/2022, 10:41 AM

Hi @millions-painter-84488, I think this may be a bug. It looks like propeller handles execution timeouts as retryable failures. However, as you noted there is no attempted cleanup of the timed out resource. Do you mind filing an issue for this? I think a best-effort abort on the resource (which will call the Delete API for web plugins as you have shown) might be the best way to handle it.

millions-painter-84488

03/29/2022, 11:46 AM

Done! https://github.com/flyteorg/flyte/issues/2298

hallowed-mouse-14616

03/29/2022, 3:57 PM

Perfect, thanks so much!

190 Views

Open in Slack

Previous Next