https://flyte.org logo
#ask-the-community
Title
# ask-the-community
h

Haytham Amin

02/25/2024, 11:42 PM
Flyte-binary chart not accepting new task_resource values:
Copy code
# inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration
  inline:
   task_resources:
      defaults:
        cpu: 2
        memory: 3Gi
        storage: 2Gi
      limits:
        memory: 3Gi
but task running showing
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
task errors out mid execution with the following error:
OOMKilled with exit code 137
y

Yee

02/26/2024, 8:25 PM
can you remove
storage
?
did you upgrade to the latest helm chart?
h

Haytham Amin

02/26/2024, 8:26 PM
Yes I’m on latest release
y

Yee

02/26/2024, 8:28 PM
delete storage
maybe it’s causing issues. apologies
that field never did anything.
there’s a different field called
ephemeralStorage
which does do something.
if you need that functionality, but storage itself never did anything
h

Haytham Amin

02/26/2024, 9:03 PM
I changed setting to # inline Specify additional configuration or overrides for Flyte, to be merged with the base configuration inline: task_resources: defaults: cpu: 2 memory: 3Gi limits: cpu: 2 memory: 3Gi but the task container still coming up with:
Copy code
name: a4dlhcql89l2zsndtzcp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
feels like there is max limit set here
when I remove the setting it goes back to 1Gi but when increase beyond 2Gi it does not reflect the change
y

Yee

02/26/2024, 9:56 PM
that should not be the case. can you make sure there are no matchable overrides
h

Haytham Amin

02/26/2024, 9:58 PM
It’s the only instance for sure! I tested by commenting it out completely and got the default values
y

Yee

02/27/2024, 6:31 PM
can you manually update the config map in k8s, restart the pods, and then try again.
just want to leave helm out of it for a moment.
and sanity check - you’re sure there’s nothing set in the task definition itself right?
h

Haytham Amin

02/27/2024, 6:32 PM
I can try that! Yes nothing in task definition
In flute-binary I had to add this as inline
Copy code
100-inline-config.yaml
storage:
  limits:
    maxDownloadMBs: 10
task_resources:
  limits:
    cpu: 2
    memory: 4Gi
  requests:
    cpu: 2
    memory: 3Gi
I dont think this will ever pickup inline task_resources limits
Copy code
var taskResourceConfig = config.MustRegisterSection(taskResourceKey, &TaskResourceSpec{
	Defaults: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("200Mi"),
	},
	Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
	},
})
y

Yee

02/28/2024, 12:47 AM
what do you mean?
that’s correct yeah
the “inline” bit is part of the helm chart, not the actual config
h

Haytham Amin

02/28/2024, 12:48 AM
its not pulling from config
y

Yee

02/28/2024, 12:48 AM
it’s confusing where the helm chart ends and flyte configuration begins.
h

Haytham Amin

02/28/2024, 12:48 AM
its the limits that are set here that are taking effect
y

Yee

02/28/2024, 12:49 AM
that’s why i was saying just edit the configmap directly on k8s
h

Haytham Amin

02/28/2024, 12:49 AM
but I cant do that
y

Yee

02/28/2024, 12:49 AM
what do you mean?
you can’t kubectl edit?
h

Haytham Amin

02/28/2024, 12:49 AM
once the task starts in the pod I am not allowed to change this param
y

Yee

02/28/2024, 12:49 AM
that’s fine
h

Haytham Amin

02/28/2024, 12:49 AM
I can put it came back with an error
y

Yee

02/28/2024, 12:49 AM
i’m saying change flyte config
kubectl edit configmap <name>
h

Haytham Amin

02/28/2024, 12:50 AM
tsk_resource is not mentioned in the deplyment.yaml
the configmap is actually correct
y

Yee

02/28/2024, 12:50 AM
can you do get configmap?
and paste here?
just the relevant part
h

Haytham Amin

02/28/2024, 12:51 AM
image.png
Copy code
kubectl edit configmap flyte-flyte-binary-config  -n flyte
100-inline-config.yaml: |
    storage:
      limits:
        maxDownloadMBs: 10
    task_resources:
      defaults:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
      limits:
        cpu: 2
        ephemeralStorage: 0
        gpu: 0
        memory: 3Gi
kind: ConfigMap
metadata:
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
  creationTimestamp: "2024-02-25T23:05:30Z"
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.10.7
  name: flyte-flyte-binary-config
  namespace: flyte
  resourceVersion: "544216950"
y

Yee

02/28/2024, 12:56 AM
so that’s correct right?
h

Haytham Amin

02/28/2024, 12:57 AM
but not reflected on the task once it runs
y

Yee

02/28/2024, 12:57 AM
and it’s still coming up with cpu 1?
h

Haytham Amin

02/28/2024, 12:57 AM
yup
Copy code
name: axknsrmtbqtttkqpv5pr-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
y

Yee

02/28/2024, 1:01 AM
i have no idea…
it’s the config, matchable resources, and the task definition itself.
h

Haytham Amin

02/28/2024, 1:03 AM
y

Yee

02/28/2024, 1:04 AM
does it?
h

Haytham Amin

02/28/2024, 1:04 AM
yes
y

Yee

02/28/2024, 1:04 AM
this says cpu is 2
not 1
h

Haytham Amin

02/28/2024, 1:04 AM
no I mean what gets passed to the task
y

Yee

02/28/2024, 1:05 AM
i thought you said what gets passed to the task was cpu = 1
h

Haytham Amin

02/28/2024, 1:05 AM
this is the task now
Copy code
resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
this is the set limits
Copy code
Limits: interfaces.TaskResourceSet{
		CPU:    resource.MustParse("2"),
		Memory: resource.MustParse("1Gi"),
		GPU:    resource.MustParse("1"),
yes task run with what seems like code defined limits
and ignores the configmap settings although they are there
i..e stored in ns
tbh I dont know where its picking up these resources configs from
removed task_resources from inline
now task runs with
Copy code
name: adq7d774wtpfs5ts78zp-n1-0
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 1Gi