Hello, good morning! We recently had an issue during registration with a large workflow (it has >100 task nodes, and calls two sub-workflows within the main workflow). The docker images build fine, and it also seemingly packages okay, but in the middle of the registration process we get a
context deadline exceeded
rpc error. It looks like this:
The error:
------------------------------------------------------------------------ --------- ---------------------------------------------------
| /tmp/register4200568562/16_workflow.cx_topic_model_workflow_2.pb | Failed | Error registering file due to rpc error: code = |
| | | DeadlineExceeded desc = context deadline exceeded |
------------------------------------------------------------------------ --------- ---------------------------------------------------
In the doc linked below, it's mentioned that the flytectl wait deadline is 15 seconds:
https://docs.flyte.org/en/latest/community/troubleshoot.html#troubles-with-flytectl-commands-with-auth-enabled
We looked around but weren't able to find the setting to increase that wait time. Any guidance here would be much appreciated!! Thanks so much
Also, here's the packaging success message (we have a 37-node wf in this specific message, but it's the same registration issue):
Packaging app_module.workflow.cx_topic_model_workflow -> 36_app_module.workflow.cx_topic_model_workflow_3.pb
Successfully packaged 37 flyte objects into /root/flyte-package.tgz