New RunLLM response generated: >>>*# Question* Can someone help me understand how a timeout set at a task level

@task(timeout=timedelta(hours=1))

interacts with the node default set in the propeller / scheduler configuration?

default-deadlines:
  node-active-deadline: 168h0m0s
  node-execution-deadline: 72h0m0s
  workflow-active-deadline: 0s

My assumption was that a task level timeout spec would override the default we set. What we've observed is that a value under 72h in the task decorator does terminate the task but anything over 72 hours still terminates the task at 72hours with the same message:

Last Error: USER::task execution timeout [72h0m0s] expired

# Answer Your observation is correct. The task-level timeout specified in the

@task

decorator does not override the

node-execution-deadline

set in the propeller configuration. The

node-execution-deadline

acts as a hard limit for task execution time, and any task running longer than this limit will be terminated, regardless of the timeout specified in the task decorator. According to Flyte's GitHub issue #3642, single task executions receive a default hard-coded timeout of 24 hours, which does not respect the default deadlines specified at the platform level or the timeout specified at the task level. This issue has been fixed in later versions of Flyte, but the

node-execution-deadline

still acts as a hard limit. To ensure that tasks can run longer than the

node-execution-deadline

, you need to increase the

node-execution-deadline

value in the propeller configuration to a value greater than the task-level timeout. Would you like more details on how to configure these settings or any other aspect of Flyte? Sources: - Flyte GitHub issue #3642 - Flyte GitHub pull request #4766 Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1721335529.588799 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.