https://flyte.org logo
r

Rahul Mehta

08/30/2022, 4:27 PM
Is this line in the docs correct? https://docs.flyte.org/en/latest/deployment/cluster_config/general.html
In the absence of an override, the global default values in the FlyteAdmin config are used.
https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L35,L43 this looked like the resource requests/limits for the flyteadmin pods themselves, is flyte actually using those as the defaults for tasks? I just tried updating
task_resource_defaults
in the helm chart w/ new limits and I didn't see the change reflected in the configmap after applying the changes to the cluster
k

katrina

08/30/2022, 4:27 PM
you mean the flyteadmin configmap didn't change after helm upgrading?
r

Rahul Mehta

08/30/2022, 4:28 PM
Yes, sorry -- the changes weren't reflected in the
flyte-admin-base-config
ConfigMap
for user-facing task resources
r

Rahul Mehta

08/30/2022, 4:30 PM
So that's what we're overriding right now: These are our overrides in
values.yaml
Copy code
task_resource_defaults:
      task_resources:
        defaults:
          cpu: 1
          memory: 1Gi
          storage: 1Gi
        limits:
          cpu: 10
          memory: 50Gi
          storage: 2Gi
Yet this is what's reflected in the configmap after a helm upgrade:
Copy code
task_resources:
  defaults:
    cpu: 100m
    memory: 500Mi
    storage: 500Mi
  limits:
    cpu: 2
    gpu: 1
    memory: 1Gi
    storage: 20Mi
k

katrina

08/30/2022, 4:34 PM
the link you sent in the first message points to a different config block though
that actually modifies the flyteadmin deployment
(like you said)
r

Rahul Mehta

08/30/2022, 4:35 PM
Sorry, think there are two separate issues I may have conflated: 1. The docs appear to suggest modifying the flyte admin configuration to set global task resource (which isn't correct as we're discussing) 2. I think we were able to resolve the issue actually, I had copied the eks values.yaml from the flyte repo, but our chart had a small bug
k

katrina

08/30/2022, 4:52 PM
ah great! would you be interested in opening a docs PR to point to the correct configuration? 😄 if not totally okay, i'll file an issue to track
r

Rahul Mehta

08/30/2022, 4:52 PM
Yep happy to do so! I think a broader issue to think through is the fact that the helm upgrade appeared to succeed but failed silently due to the indentation problem
So all our gitops automation was green despite the bug
k

katrina

08/30/2022, 4:54 PM
oh, that is certainly not great. i wonder if there's a helm "strict mode" or equivalent
r

Rahul Mehta

08/30/2022, 5:49 PM
Also the indentation issue also led to us having only 4 workers instead of 40 in propeller...fixing that is leading to much better perf
k

katrina

08/30/2022, 6:27 PM
wow!
just to clarify, was this an indentation issue on your end or our templates?
r

Rahul Mehta

08/30/2022, 6:28 PM
On my end 😅
35 Views