Hello, good morning! We recently had an issue duri...
# ask-the-community
s
Hello, good morning! We recently had an issue during registration with a large workflow (it has >100 task nodes, and calls two sub-workflows within the main workflow). The docker images build fine, and it also seemingly packages okay, but in the middle of the registration process we get a
context deadline exceeded
rpc error. It looks like this: The error:
Copy code
------------------------------------------------------------------------ --------- --------------------------------------------------- 
| /tmp/register4200568562/16_workflow.cx_topic_model_workflow_2.pb       | Failed  | Error registering file due to rpc error: code =   |
|                                                                        |         | DeadlineExceeded desc = context deadline exceeded |
 ------------------------------------------------------------------------ --------- ---------------------------------------------------
In the doc linked below, it's mentioned that the flytectl wait deadline is 15 seconds: https://docs.flyte.org/en/latest/community/troubleshoot.html#troubles-with-flytectl-commands-with-auth-enabled We looked around but weren't able to find the setting to increase that wait time. Any guidance here would be much appreciated!! Thanks so much Also, here's the packaging success message (we have a 37-node wf in this specific message, but it's the same registration issue):
Copy code
Packaging app_module.workflow.cx_topic_model_workflow -> 36_app_module.workflow.cx_topic_model_workflow_3.pb
Successfully packaged 37 flyte objects into /root/flyte-package.tgz
k
100 nodes is not large. I think this is a timeout
So one of the underlying services is slower than expected- s3/db/auth
15 seconds is a lot
Rather an eternity
The overall error is a red herring, can you check admin logs and help improve the end user message
s
Ok thanks @Ketan (kumare3), looking into this
Update on this: we realized that looping within the workflow to create tasks is producing the error -- when we only created one iterations-worth of tasks, we don't get the error. Should we be looping within a workflow to create tasks? Or what would be the best way to do something similar within the workflow context?
s
Are you looping in a dynamic workflow?
k
@Samhita Alla I think they are creating the workflow statically
and that is resulting in a too big of a workflow. i did not see the code so guessing
110 Views