check admin memory etc

Currently we just have one flyteadmin pod but it never gets anywhere near its resource requests 1 CPU core 1GiB of memory. Does flyteadmin have much parallelism within a single process? Probably we should create multiple smaller replicas as was mentioned on https://docs.flyte.org/en/latest/deployment/configuration/performance.html#scaling-out-flyteadmin

but what is happening maybe is that the database writes are locked

but i need to see more

actually there is some performance stuff already done by @hallowed-mouse-14616 and it is available

@flat-exabyte-79377 / @calm-pilot-2010 would also love to see the workflow sample too

maybe you are running over etcD limit and offloading maybe a good idea

Actually I do see some etcD request too large errors that are definitely relevant.

{"json":{"exec_id":"anbwtd2v2x5p8px67rzt","ns":"nerf-test-production","routine":"worker-672"},"level":"error","msg":"Failed to update workflow. Error [etcdserver: request is too large]","ts":"2023-11-09T11:37:18Z"}
E1109 11:37:18.581775       1 workers.go:103] error syncing '<project-domain>/anbwtd2v2x5p8px67rzt': etcdserver: request is too large

From the metrics etcd does not look particularly healthy

For reference the workflow structure is like this

This version is more representative

If I fetch one of the big CRDs using

kubectl get flyteworkflows f0319e66fb12740eaa4d -o yaml > etcd_sync_failure.yaml

the file size is

1.9MB

. If I use a python script to delete all the

message

fields the size is only

0.38MB

. I think its quite likely that tracebacks in the CRDs are pushing them over the edge to be too big.

This use-case is a bit crazy I know but it is the easiest way for other teams at our company to go from python function that works somewhat reliably to running it at really big scale and getting some valuable results. Ideally they would not need to learn about any distributed compute framework. It seems so me that the size of CRDs is definitely the biggest scalability issue with running flyte workflows like this with really big fan-out. As far as I understand the

useOffloadedWorkflowClosure

option won'h help us much because the majority of the CRD size is non-static given our use of dynamic workflows. Is there any possibility of improving this? For example I tried hacking something together to preventing the tracebacks from getting into the CRD. That helped a bit but the resulting CRD was still much larger than one where everything succeeded and it probably only allows maybe 4X greater scalability (based on the CRD size numbers I got above). Would it be possible to store the CRD in a more compressed format? My best guess is that currently the CRD data is stored as YAML or JSON but if it was stored as a protobuf probably the size would be much smaller.

Sadly we will have to dig deeper into this and at the moment a little busy. Will get to it hopefully sometime

I actually just ran the test with the offload enabled. Unfortunately it doesn't seem to have helped much 🙁. From a quick look it seems like the only difference is that the

spec

field is

null

when using offload but in this case that is quite a small portion of the total CRD. I think we will need to come up with a solution for this. Possibly we could make a contribution to flyte. Would you mind explaining why etcd is used for storing this. What stuff updates the CRD? Is it just flytepropeller or does flyteadmin also make updates? Would it be feasible to use something like a k8s persistent volume claim? Its quite likely I will experiment with some of my ideas so if you can provide any insight on whether they are feasible that would be much appreciated 🙂.

Also we plan to compress further - cc @high-park-82026 / @hallowed-mouse-14616

@calm-pilot-2010 i was looking into it some today, this is the suggestion i had given some time ago - haha so couple question • what is the chunk size? • what is the size of

l

is map task ideal? map tasks are designed to automatically compress the storage usage in etcd

@calm-pilot-2010 i think we should hop on a call

I ended up making a couple of alterations to flytepropeller: https://github.com/Tom-Newton/flyte/pull/2 and https://github.com/Tom-Newton/flyte/pull/3. These implement point 1 and they seem to be very effective. I'm hoping we could upstream something along these lines if we put them behind a feature flag.

I think those changes resolve the etcd size limit problems, but I think that is actually a different problem, to the

context deadline exceeded

errors we saw first. I'm reasonably confident these

context deadline exceeded

errors are coming from networking issues between our control plane cluster and the data plane cluster. When I run the same workflow all in the same cluster as the control plane I haven't been able to reproduce the issue. I'm hoping that scaling up our nginx ingress controller will resolve the context deadline exceeded errors.

Thanks for looking at my PRs @hallowed-mouse-14616. If we think its somewhat sensible then I can try to make versions that could by up-streamed, with feature flags etc.

i'm speaking for a lot of flyte users, but really appreciate the effort here. some of the stuff you've hacked on is certainly in the weeds 🙌