https://flyte.org
Join Slack
Flyte-binary chart not accepting new task_resource values: ```# inline Specify additional configurat...
# flyte-support
c
Flyte-binary chart not accepting new task_resource values:
Copy code
# inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration
  inline:
   task_resources:
      defaults:
        cpu: 2
        memory: 3Gi
        storage: 2Gi
      limits:
        memory: 3Gi
but task running showing
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
task errors out mid execution with the following error: 
OOMKilled with exit code 137
t
can you remove 
storage
?
did you upgrade to the latest helm chart?
c
Yes I’m on latest release
t
delete storage
maybe it’s causing issues. apologies
that field never did anything.
there’s a different field called 
ephemeralStorage
which does do something.
if you need that functionality, but storage itself never did anything
👍 1
c
I changed setting to # inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration inline: task_resources: defaults: cpu: 2 memory: 3Gi limits: cpu: 2 memory: 3Gi but the task container still coming up with:
Copy code
name: a4dlhcql89l2zsndtzcp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
feels like there is max limit set here
when I remove the setting it goes back to 1Gi but when increase beyond 2Gi it does not reflect the change
t
that should not be the case. can you make sure there are no matchable overrides
c
It’s the only instance for sure! I tested by commenting it out completely and got the default values
t
can you manually update the config map in k8s, restart the pods, and then try again.
just want to leave helm out of it for a moment.
and sanity check - you’re sure there’s nothing set in the task definition itself right?
c
I can try that! Yes nothing in task definition
In flute-binary I had to add this as inline
Copy code
100-inline-config.yaml
storage:
  limits:
    maxDownloadMBs: 10
task_resources:
  limits:
    cpu: 2
    memory: 4Gi
  requests:
    cpu: 2
    memory: 3Gi
I dont think this will ever pickup inline task_resources limits
Copy code
var taskResourceConfig = config.MustRegisterSection(taskResourceKey, &TaskResourceSpec{
	Defaults: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("200Mi"),
	},
	Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
	},
})
t
what do you mean?
that’s correct yeah
the “inline” bit is part of the helm chart, not the actual config
c
its not pulling from config
t
it’s confusing where the helm chart ends and flyte configuration begins.
c
its the limits that are set here that are taking effect
t
that’s why i was saying just edit the configmap directly on k8s
c
but I cant do that
t
what do you mean?
you can’t kubectl edit?
c
once the task starts in the pod I am not allowed to change this param
t
that’s fine
c
I can put it came back with an error
t
i’m saying change flyte config
kubectl edit configmap <name>
c
tsk_resource is not mentioned in the deplyment.yaml
the configmap is actually correct
t
can you do get configmap?
and paste here?
just the relevant part
c
Copy code
kubectl edit configmap flyte-flyte-binary-config  -n flyte
100-inline-config.yaml: |
    storage:
      limits:
        maxDownloadMBs: 10
    task_resources:
      defaults:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
      limits:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
kind: ConfigMap
metadata:
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
  creationTimestamp: "2024-02-25T23:05:30Z"
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.10.7
  name: flyte-flyte-binary-config
  namespace: flyte
  resourceVersion: "544216950"
t
so that’s correct right?
c
but not reflected on the task once it runs
t
and it’s still coming up with cpu 1?
c
yup
Copy code
name: axknsrmtbqtttkqpv5pr-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
t
i have no idea…
it’s the config, matchable resources, and the task definition itself.
c
t
does it?
c
yes
t
this says cpu is 2
not 1
c
no I mean what gets passed to the task
t
i thought you said what gets passed to the task was cpu = 1
c
this is the task now
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
this is the set limits
Copy code
Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
yes task run with what seems like code defined limits
and ignores the configmap settings although they are there
i..e stored in ns
tbh I dont know where its picking up these resources configs from
removed task_resources from inline
now task runs with
Copy code
name: adq7d774wtpfs5ts78zp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
23 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by