Rahul Mehta

    Rahul Mehta

    3 weeks ago
    Is this line in the docs correct? https://docs.flyte.org/en/latest/deployment/cluster_config/general.html
    In the absence of an override, the global default values in the FlyteAdmin config are used.
    https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L35,L43 this looked like the resource requests/limits for the flyteadmin pods themselves, is flyte actually using those as the defaults for tasks? I just tried updating
    task_resource_defaults
    in the helm chart w/ new limits and I didn't see the change reflected in the configmap after applying the changes to the cluster
    k

    katrina

    3 weeks ago
    you mean the flyteadmin configmap didn't change after helm upgrading?
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    Yes, sorry -- the changes weren't reflected in the
    flyte-admin-base-config
    ConfigMap
    for user-facing task resources
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    So that's what we're overriding right now: These are our overrides in
    values.yaml
    task_resource_defaults:
          task_resources:
            defaults:
              cpu: 1
              memory: 1Gi
              storage: 1Gi
            limits:
              cpu: 10
              memory: 50Gi
              storage: 2Gi
    Yet this is what's reflected in the configmap after a helm upgrade:
    task_resources:
      defaults:
        cpu: 100m
        memory: 500Mi
        storage: 500Mi
      limits:
        cpu: 2
        gpu: 1
        memory: 1Gi
        storage: 20Mi
    k

    katrina

    3 weeks ago
    the link you sent in the first message points to a different config block though
    that actually modifies the flyteadmin deployment
    (like you said)
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    Sorry, think there are two separate issues I may have conflated:1. The docs appear to suggest modifying the flyte admin configuration to set global task resource (which isn't correct as we're discussing) 2. I think we were able to resolve the issue actually, I had copied the eks values.yaml from the flyte repo, but our chart had a small bug
    k

    katrina

    3 weeks ago
    ah great! would you be interested in opening a docs PR to point to the correct configuration? 😄 if not totally okay, i'll file an issue to track
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    Yep happy to do so! I think a broader issue to think through is the fact that the helm upgrade appeared to succeed but failed silently due to the indentation problem
    So all our gitops automation was green despite the bug
    k

    katrina

    3 weeks ago
    oh, that is certainly not great. i wonder if there's a helm "strict mode" or equivalent
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    Also the indentation issue also led to us having only 4 workers instead of 40 in propeller...fixing that is leading to much better perf
    k

    katrina

    3 weeks ago
    wow!
    just to clarify, was this an indentation issue on your end or our templates?
    Rahul Mehta

    Rahul Mehta

    3 weeks ago
    On my end 😅