Hi all,
we are currently using Flyte in version 1....
# ask-the-communityt
Tim Hebbeler
08/16/2023, 9:10 AMHi all,
we are currently using Flyte in version 1.9 (exits in previous versions as well) and have problems when we feed the "map_task" with a large number of inputs (~5000). It is then no longer possible for us to see the inputs in flyteconsole. In addition, flyteadmin crashes regularly when you want to display the structure or the inputs in flyteadmin. Is the problem known?
I think it is related to this issue:
https://github.com/flyteorg/flyte/issues/3337
We use Postgresql as the database. I do not see that some limit in memory is reached.
Thanks a lot!
k
Ketan (kumare3)
08/16/2023, 1:25 PM
Cc @katrina / @Prafulla Mahindrakar fyi
Ketan (kumare3)
08/16/2023, 1:26 PM
Can you share specifics on the 5k list? How large is each item - are you running the pod on very low memory?
Ketan (kumare3)
08/16/2023, 1:27 PM
Also @Tim Hebbeler did the workflow complete on propeller?
t
Tim Hebbeler
08/16/2023, 1:45 PMwe already increase grpc message limit on the backend. Propeller completes the workflow.
each element consists of 3 paths to the cloud storage. A path ~150 characters. We also have bound inputs that require memory
what is the general suggestion when the grpc message gets so big?
k
Ketan (kumare3)
08/16/2023, 2:01 PM
Hmm we prefer it should not get this big
Ketan (kumare3)
08/16/2023, 2:01 PM
You should breakdown the map task into multiple
t
Tim Hebbeler
08/16/2023, 2:26 PMwe now increase maxMessageSizeBytes to 20mb, but flyteadmin is still restarting.
Liveness probe failed:
k
Ketan (kumare3)
08/16/2023, 2:30 PM
This is something else
Ketan (kumare3)
08/16/2023, 2:31 PM
The total payload for aggregated state of a map task is probably too high
Ketan (kumare3)
08/16/2023, 2:31 PM
This is why workflow completed - but admin view services fail
Ketan (kumare3)
08/16/2023, 2:31 PM
Cc @Dan Rammer (hamersaw) let’s talk about this
Ketan (kumare3)
08/16/2023, 2:32 PM
@Tim Hebbeler would you be open to hopping on a call to us to see what the problem is - we need more debug information
Ketan (kumare3)
08/16/2023, 2:33 PM
I am actually trying to figure if the problem is due to inputs or outputs or state?
t
Tim Hebbeler
08/16/2023, 2:38 PMyes, crash happens when i open the Flyte Console on a task and open the Inputs tab. Maybe flyteadmin is to busy to answer the request. I can also see some slow SQL query inside flyteadmin.
2023/08/16 14:34:15 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134 SLOW SQL >= 200msk
Ketan (kumare3)
08/16/2023, 2:38 PM
Wow
Ketan (kumare3)
08/16/2023, 2:39 PM
Also FlyteAdmin memory?
t
Tim Hebbeler
08/16/2023, 2:40 PMlimit is set to 1Gi
k
Ketan (kumare3)
08/16/2023, 2:48 PM
I think that’s your problem
Ketan (kumare3)
08/16/2023, 2:48 PM
It’s probably bursting to 2/4G
Ketan (kumare3)
08/16/2023, 2:48 PM
Can you try increasing it
t
Tim Hebbeler
08/16/2023, 3:05 PMthanks a lot increasing the memory to 4Gi for flyteadmin helps
Tim Hebbeler
08/16/2023, 3:10 PMfetch of map_task inputs in Flyte Console is still slow. Maybe pagination would help
35 Views