https://flyte.org
Join Slack
Hi all, we are currently using Flyte in version 1.9 (exits in previous versions as well) and have pr...
# flyte-support
n
Hi all, we are currently using Flyte in version 1.9 (exits in previous versions as well) and have problems when we feed the "map_task" with a large number of inputs (~5000). It is then no longer possible for us to see the inputs in flyteconsole. In addition, flyteadmin crashes regularly when you want to display the structure or the inputs in flyteadmin. Is the problem known? I think it is related to this issue: https://github.com/flyteorg/flyte/issues/3337 We use Postgresql as the database. I do not see that some limit in memory is reached. Thanks a lot!
f
Cc @acceptable-policeman-57188 / @icy-agent-73298 fyi
Can you share specifics on the 5k list? How large is each item - are you running the pod on very low memory?
Also @nice-article-27102 did the workflow complete on propeller?
n
we already increase grpc message limit on the backend. Propeller completes the workflow. each element consists of 3 paths to the cloud storage. A path ~150 characters. We also have bound inputs that require memory what is the general suggestion when the grpc message gets so big?
f
Hmm we prefer it should not get this big
You should breakdown the map task into multiple
n
we now increase maxMessageSizeBytes to 20mb, but flyteadmin is still restarting. Liveness probe failed:
f
This is something else
The total payload for aggregated state of a map task is probably too high
This is why workflow completed - but admin view services fail
Cc @hallowed-mouse-14616 let’s talk about this
👍 1
@nice-article-27102 would you be open to hopping on a call to us to see what the problem is - we need more debug information
I am actually trying to figure if the problem is due to inputs or outputs or state?
n
yes, crash happens when i open the Flyte Console on a task and open the Inputs tab. Maybe flyteadmin is to busy to answer the request. I can also see some slow SQL query inside flyteadmin. 
2023/08/16 14:34:15 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134 SLOW SQL >= 200ms
*`[205.077ms] [rows:1] SELECT * FROM "tasks" WHERE "tasks"."project" = 'flyte' AND "tasks"."domain" = 'development' AND "tasks"."name" = '...' AND "tasks"."version" = 'dvspDYpCSnG9Kh3-tEKKgQ==' LIMIT 1`*
f
Wow
Also FlyteAdmin memory?
n
limit is set to 1Gi
f
I think that’s your problem
It’s probably bursting to 2/4G
Can you try increasing it
n
thanks a lot increasing the memory to 4Gi for flyteadmin helps
fetch of map_task inputs in Flyte Console is still slow. Maybe pagination would help
45 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by