Hi everyone, I am trying to update the node group instance type through env.yaml while deploying with opta. Currently it's at default t3.medium. However, adding node_instance_type with the desired instance type seems to have no effect. I am adding this under - type: k8s-cluster. Do I need to create a separate node group using opta env to achieve this?

@Yuvraj, can you help?

@Harshit Sharma yes that is correct, Can you post the output of

opta apply

?

Hi, @Yuvraj I was able to resolve the issue. Thanks for the information though.

I don’t think data size matter, Can you post your example here cc: @Samhita Alla

Sure let me get the SS of the code. let me know if you need anything else.

@Harshit Sharma, can you set resource requests on your first task just in case it’s getting stuck due to resource constraints?

from flytekit import Resources, task

@task(requests=Resources(cpu="2", mem="1000Mi"))
def fetch_domain_list_df(...):
    ...

@Samhita Alla The pod being generated to undertake the task shows a requested resource of 7000Mi already, which happens to also be the limit for task resources.

Can you also check the pod logs even if it’s successful?

yes that's showing success. For now we were able to work around this using pandas df schema provided by flyte and combining task 1 and 2.

@Dan Rammer (hamersaw), can you help? @Harshit Sharma, can you share with us the full pod log?

From the event logs, it seems like the pod is getting terminated due to pre-emption.

Found a related issue: https://github.com/flyteorg/flyte/issues/2788.

@Harshit Sharma I think there are few things we can work with here: (1) Are you using map task with a list of 1 million inputs? This would result in 1 million Pods being executed - this is beyond the capacity of k8s, are you using some sort of batching mechanism? (2) Flyte has an

inject-finalizer

configuration option which uses k8s finalizers to stop external systems from deleting the Pods, but some external systems do not respect this scheme and delete anyway. So this is a "best effort" by Flyte to keep Pods around, if they are still being deleted then unforunately there is not much we can do to stop it. A few ideas here are to look at the k8s logs to determine what is deleting the pods, additionally you can reduce the concurrency configuration on the map task so fewer Pods are spawned (typically these are deleted by some sort of resource manager). (3) The issue above (2788) has a submitted fix - but it resulted in a panic on externally deleted pods during interruptible failures. Are you designating the task as "interruptible"?

We are actually passing splitted lists as input and not each element. That contains map tasks to the number of split.

Oh OK, do you know how many splits there are?

Yeah actually, we are splitting them in 15 parts

@Harshit Sharma, have you been able to resolve the issue?

@Samhita Alla will be trying out the sol soon

You may have to modify in https://github.com/flyteorg/flyte/blob/091cb93606cbb0a62b6d4f962e0afae021616101/charts/flyte-core/values.yaml#L639-L646 section.

k8s:
  inject-finalizer: true
  ...