Hi, since updating the flyte-binary chart from 1.6.2 to 1.8.1, a lot of my tasks are killed with the following message:

code:"Interrupted" message:"The node was low on resource: ephemeral-storage. Container alxkcdhndgnqd6kzbq95-n0-0-n23-0-40 was using 20205640Ki, which exceeds its request of 0

With the previous version, the pods could use all the storage available on the node. Did allocation of ephemeral storage change between these versions? Thanks for help

This has to do with k8s version as well. Can you tell us did you change k8s version? If not it seems ephemeral storage is being set to 0 instead of ignoring. Cc @jeev / @Dan Rammer (hamersaw)

afaik, this only happens when the node is under disk pressure. looks like the task is using ~20Gi. how much storage does the node have, and are there other workloads running on the same node?

Thanks for the replies. It is possible that these errors are related to disk pressure, I was confused, since in the previous version the error message in that case was "DiskPressure". Also it seems like the old DiskPressure error also still appears. Is it possible, that the error message is different for a map_task and a python task, but the origin is the same for both - killed by kubernetes due to disk pressure?

@Klemens Kasseroller: in the previous version, the disk pressure doesn’t cause eviction?

It did, only the error message was different

What was the error?