https://flyte.org logo
Join Slack
Powered by
Hello. I have been evaluating Flyte, and I have a ...
# flyte-support
g
Hello. I have been evaluating Flyte, and I have a few questions. I have deployed Flyte in single binary mode and installed flytectl. • How can I update domain settings? Is there any documentation for all available domain settings? I have attached a screenshot for you to look over. • Flyte adds development, staging and production domains to a newly created project. Is there any way to customize this behaviour? Can I define my own domain and settings? • For every project and domain combination, Flyte creates a namespace. Can I customize this behaviour? Can I specify a namespace at the registration (
flytectl register files
) time? • I was unable to use 
accelerator=flytekit.extras.accelerators.T4
because we use Karpenter to provision Nodes, and the node selectors applied by 
flytekit.extras.accelerators.T4
did not match with what Karpenter expected. Can we define custom accelerators that can apply developer-defined node selectors and tolerations? For now, I used 
PodTemplate
. • It will be nice to have some documentation about how to structure Python modules when using 
ImageSpec
.
> Can we define custom accelerators that can apply developer defined custom node selectors and tolerations? For toleration, I used the resource toleration settings in the plugin config.
Copy code
inline:
      plugins:
        k8s:
          resource-tolerations:
          - nvidia.com/gpu:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
f
👍 1
t
Flyte adds development, staging and production domains to a newly created project. Is there any way to customize this behaviour? Can I define my own domain and settings?
i believe it should be possible to change the domain names in the backend config.
Can I specify a namespace at the registration (
flytectl register files
) time?
you can specify the project and domain you want to register your tasks or workflows in.
It will be nice to have some documentation about how to structure Python modules when using 
ImageSpec
.
+1 it'd be nice if you could create an issue and include all the docs we can add. are you seeing any issues now?
a
> Can I define my own domain You can set the list of domains in the Helm chart:
Copy code
configuration:
  inline:
    domains:
      - id: domain1
        name: domain1
      - id: domain2
        name: domain2
      - id: domain3
        name: domain3
How can I update domain settings?
The settings you see in the UI are under the 
flyteadmin
section of the config (docs) you can change it under 
configuration.inline
in the Helm chart
👍 1
did not match with what Karpenter expected
how different is what Karpenter expects? According to the examples in the PR, what you set in Propeller's config should land on the 
nodeSelectorTerms
g
> According to the examples in the PR, what you set in Propeller's config should land on the 
nodeSelectorTerms
Let me look over the PR and see if I can understand the required configuration. I did not use a custom configuration while testing. Could you please point me to the example config sections on this PR? > how different is what Karpenter expects? For T4 GPU, I can use these documented labels in the node selector. https://karpenter.sh/docs/reference/instance-types/#g4dn-family
> +1 it'd be nice if you could create an issue and include all the docs we can add. are you seeing any issues now? I will try to open an issue by today.
> You can set the list of domains in the Helm chart: > configuration: > inline: > domains: > - id: domain1 > name: domain1 > - id: domain2 > name: domain2 > - id: domain3 > name: domain3 If I understand correctly, using this configuration, I cannot customize domains per project. I cannot set up domains D1 and D2 for project P1 and D3 and D4 for project P2. Flyte automatically creates namespaces with the pattern project_name-domain_name for every combination. Can I customize this behaviour? For example, when using Argo Workflow, I can apply the Argo Workflow manifest in a namespace, and all the DAG node pods will run in that namespace.
> According to the examples in the PR, what you set in Propeller's config should land on the nodeSelectorTerms I see that I can configure the node selector label key,
Copy code
plugins:
  k8s:
    gpu-device-node-label: cloud.google.com/gke-accelerator
It seems like the value,
nvidia-tesla-t4
, is hardcoded for 
accelerator=T4
. Looking at the label values that Karpenter supports, this hardcoded value will not work with Karpenter.
> It seems like the value,nvidia-tesla-t4, is hardcoded for accelerator=T4. Looking at the label values that Karpenter supports, this hardcoded value will not work with Karpenter. I was able to get it to work by adding 
gpu-device-node-label: karpenter.k8s.aws/instance-gpu-name
in the K8s plugin config and using 
accelerator=flytekit.extras.accelerators.GPUAccelerator("t4")
in the task decorator.
🙌🏽 1
🙌 1
a
@green-crayon-74257 that's great to know and thanks for sharing. I don't think the keys for specific devices are hardcoded btw
👍 1
g
a
Flyte automatically creates namespaces with the pattern project_name-domain_name for every combination. Can I customize this behaviour?
I think the closest you can get is using the `namespace_mapping`block (that you can set under 
configuration.inline
) You can change the template Flyte uses to name the namespaces (
project-domain
) and you can even set Flyte executions to a specific namespace by configuring 
mapping
to a static value https://docs.flyte.org/en/latest/deployment/configuration/generated/flyteadmin_config.html#section-namespace-mapping But once the project and domains are defined, I don't see a way to prevent that Flyte creates all the combinations
👍 1
22 Views
Open in Slack
PreviousNext
Powered by