https://flyte.org logo
Join Slack
Powered by
Flyte-binary chart not accepting new task_resource...
# ask-the-community
h
Flyte-binary chart not accepting new task_resource values:
Copy code
# inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration
  inline:
   task_resources:
      defaults:
        cpu: 2
        memory: 3Gi
        storage: 2Gi
      limits:
        memory: 3Gi
but task running showing
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
task errors out mid execution with the following error: 
OOMKilled with exit code 137
y
can you remove 
storage
?
did you upgrade to the latest helm chart?
h
Yes I’m on latest release
y
delete storage
maybe it’s causing issues. apologies
that field never did anything.
there’s a different field called 
ephemeralStorage
which does do something.
if you need that functionality, but storage itself never did anything
h
I changed setting to # inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration inline: task_resources: defaults: cpu: 2 memory: 3Gi limits: cpu: 2 memory: 3Gi but the task container still coming up with:
Copy code
name: a4dlhcql89l2zsndtzcp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
feels like there is max limit set here
when I remove the setting it goes back to 1Gi but when increase beyond 2Gi it does not reflect the change
y
that should not be the case. can you make sure there are no matchable overrides
h
It’s the only instance for sure! I tested by commenting it out completely and got the default values
y
can you manually update the config map in k8s, restart the pods, and then try again.
just want to leave helm out of it for a moment.
and sanity check - you’re sure there’s nothing set in the task definition itself right?
h
I can try that! Yes nothing in task definition
In flute-binary I had to add this as inline
Copy code
100-inline-config.yaml
storage:
  limits:
    maxDownloadMBs: 10
task_resources:
  limits:
    cpu: 2
    memory: 4Gi
  requests:
    cpu: 2
    memory: 3Gi
I dont think this will ever pickup inline task_resources limits
Copy code
var taskResourceConfig = config.MustRegisterSection(taskResourceKey, &TaskResourceSpec{
	Defaults: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("200Mi"),
	},
	Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
	},
})
y
what do you mean?
that’s correct yeah
the “inline” bit is part of the helm chart, not the actual config
h
its not pulling from config
y
it’s confusing where the helm chart ends and flyte configuration begins.
h
its the limits that are set here that are taking effect
y
that’s why i was saying just edit the configmap directly on k8s
h
but I cant do that
y
what do you mean?
you can’t kubectl edit?
h
once the task starts in the pod I am not allowed to change this param
y
that’s fine
h
I can put it came back with an error
y
i’m saying change flyte config
kubectl edit configmap <name>
h
tsk_resource is not mentioned in the deplyment.yaml
the configmap is actually correct
y
can you do get configmap?
and paste here?
just the relevant part
h
image.png
Copy code
kubectl edit configmap flyte-flyte-binary-config  -n flyte
100-inline-config.yaml: |
    storage:
      limits:
        maxDownloadMBs: 10
    task_resources:
      defaults:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
      limits:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
kind: ConfigMap
metadata:
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
  creationTimestamp: "2024-02-25T23:05:30Z"
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.10.7
  name: flyte-flyte-binary-config
  namespace: flyte
  resourceVersion: "544216950"
y
so that’s correct right?
h
but not reflected on the task once it runs
y
and it’s still coming up with cpu 1?
h
yup
Copy code
name: axknsrmtbqtttkqpv5pr-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
y
i have no idea…
it’s the config, matchable resources, and the task definition itself.
h
y
does it?
h
yes
y
this says cpu is 2
not 1
h
no I mean what gets passed to the task
y
i thought you said what gets passed to the task was cpu = 1
h
this is the task now
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
this is the set limits
Copy code
Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
yes task run with what seems like code defined limits
and ignores the configmap settings although they are there
i..e stored in ns
tbh I dont know where its picking up these resources configs from
removed task_resources from inline
now task runs with
Copy code
name: adq7d774wtpfs5ts78zp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi
12 Views
Open in Slack
PreviousNext
Powered by