Debajyoti Chatterjee
02/12/2024, 10:56 AMflytectl register files
) time?
• I was unable to use accelerator=flytekit.extras.accelerators.T4
because we use Karpenter to provision Nodes, and the node selectors applied by flytekit.extras.accelerators.T4
did not match with what Karpenter expected. Can we define custom accelerators that can apply developer-defined node selectors and tolerations? For now, I used PodTemplate
.
• It will be nice to have some documentation about how to structure Python modules when using ImageSpec
.Debajyoti Chatterjee
02/12/2024, 11:03 AMinline:
plugins:
k8s:
resource-tolerations:
- nvidia.com/gpu:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
Ketan (kumare3)
Samhita Alla
Flyte adds development, staging and production domains to a newly created project. Is there any way to customize this behaviour? Can I define my own domain and settings?i believe it should be possible to change the domain names in the backend config.
Can I specify a namespace at the registration (you can specify the project and domain you want to register your tasks or workflows in.) time?flytectl register files
It will be nice to have some documentation about how to structure Python modules when using+1 it'd be nice if you could create an issue and include all the docs we can add. are you seeing any issues now?.ImageSpec
David Espejo (he/him)
02/12/2024, 5:21 PMconfiguration:
inline:
domains:
- id: domain1
name: domain1
- id: domain2
name: domain2
- id: domain3
name: domain3
David Espejo (he/him)
02/12/2024, 5:23 PMHow can I update domain settings?The settings you see in the UI are under the
flyteadmin
section of the config (docs)
you can change it under configuration.inline
in the Helm chartDavid Espejo (he/him)
02/12/2024, 5:27 PMdid not match with what Karpenter expectedhow different is what Karpenter expects? According to the examples in the PR, what you set in Propeller's config should land on the
nodeSelectorTerms
Debajyoti Chatterjee
02/13/2024, 3:39 AMnodeSelectorTerms
Let me look over the PR and see if I can understand the required configuration. I did not use a custom configuration while testing. Could you please point me to the example config sections on this PR?
> how different is what Karpenter expects?
For T4 GPU, I can use these documented labels in the node selector. https://karpenter.sh/docs/reference/instance-types/#g4dn-familyDebajyoti Chatterjee
02/13/2024, 4:15 AMDebajyoti Chatterjee
02/13/2024, 4:21 AMDebajyoti Chatterjee
02/13/2024, 4:30 AMplugins:
k8s:
gpu-device-node-label: cloud.google.com/gke-accelerator
It seems like the value,nvidia-tesla-t4
, is hardcoded for accelerator=T4
. Looking at the label values that Karpenter supports, this hardcoded value will not work with Karpenter.Debajyoti Chatterjee
02/13/2024, 7:53 AMgpu-device-node-label: karpenter.k8s.aws/instance-gpu-name
in the K8s plugin config and using accelerator=flytekit.extras.accelerators.GPUAccelerator("t4")
in the task decorator.David Espejo (he/him)
02/13/2024, 3:31 PMDebajyoti Chatterjee
02/13/2024, 3:56 PMDavid Espejo (he/him)
02/13/2024, 4:01 PMFlyte automatically creates namespaces with the pattern project_name-domain_name for every combination. Can I customize this behaviour?I think the closest you can get is using the `namespace_mapping`block (that you can set under
configuration.inline
)
You can change the template Flyte uses to name the namespaces (project-domain
) and you can even set Flyte executions to a specific namespace by configuring mapping
to a static value
https://docs.flyte.org/en/latest/deployment/configuration/generated/flyteadmin_config.html#section-namespace-mapping
But once the project and domains are defined, I don't see a way to prevent that Flyte creates all the combinations