I couldn’t find this, but is there a way to specify the machine type in the task? I see options for cpu, gpu, and mem, but not for which types. I think we could train using AWS Sagemaker where the machine type in configurable, but seems unnecessary if I can just do it in Flyte. Let me know what I am missing! Thank you

so Flyte folllows K8s philosophy of abstracting away specifics about compute like machine type of the underlying cloud interface, hence options for cpu, gpu, mem. The AWS sagemaker integration allows for this (see the

SagemakerTrainingJobConfig

here) @Kevin Su @Samhita Alla not sure if you input here on how to configure machine types at the Flyte cluster configuration level.

Ah okay. I think Kubeflow lets you do this and it, at least to me, it is a pretty important feature for running jobs natively. Otherwise, I might need databricks and sagemaker to get the optimized compute machines. It is possible that the speed up is negligible for our jobs, but for some jobs it could make a big difference

what machine type are you trying to specify?

I think it would be the m5 and c5 types. I honestly am not sure what our cluster runs on, but I will find out.

Requests and limits are the mechanisms Kubernetes uses to control resources IIRC, we don’t support specify the machine type in the task for now. cc @Ketan (kumare3) @Dan Rammer (hamersaw) is there any other way to run the task on a specific machine type?

Ah okay yeah this looks like vertex ai doesn’t do this either. They will choose the appropriate type depending on the job. I don’t really know what that means, but does Flyte do something similar if you request like 96 cpus?

@Evan Sadler how can kubeflow do it - I do not know how to do it in K8s - I think you might want to choose the right node-pools and that can be done using

labels and annotations

and or node selectors

@Ketan (kumare3) I am speaking from an area of very little experience so I apologize. I think we setup multiple node pools and picked between them. I will double check with my old colleagues and see.

absolutely, ohh no harm. I just wanted to explain my ignorance

Another thought is taints / tolerations. Not sure if there is anyway to interface with K8s machine types there. May be worth looking into.