https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
n

Nan Qin

03/31/2023, 4:41 PM
is it possible to run a workflow in a different namespace other than 
project-domain
?
found docs about namespace_mapping, are there docs/guides on how to change and apply those configurations in general?
j

Jay Ganbat

03/31/2023, 5:33 PM
maybe reference LP/Workflow is useful here https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.reference_workflow.html#flytekit-reference-workflow if you are calling it from different project without config changes
k

Ketan (kumare3)

04/01/2023, 12:21 AM
When you launch a workflow, the execution domain is different from the original domain and project
You do not need to use reference workflows, just simply launch an execution in a different project and domain
n

Nan Qin

04/01/2023, 1:00 AM
is it possible to launch executions in a name space not following the 
project-domain
convention, eg a namespace called 
experiments
?
k

Ketan (kumare3)

04/01/2023, 1:20 AM
Ohh are you talking about k8s namespaces
If so yes
n

Nan Qin

04/01/2023, 1:20 AM
yes k8 namespaces
k

Ketan (kumare3)

04/01/2023, 1:20 AM
K8s namespaces are independent from project domain, default is to create project-domain namespaces
You will Have to configure
The backend for this
n

Nan Qin

04/01/2023, 1:22 AM
namespace_mapping and template?
k

Ketan (kumare3)

04/01/2023, 1:23 AM
Yup I think so - AFK, friday evening 😬
n

Nan Qin

04/01/2023, 1:26 AM
no worries, let's continue when you have time have a good weekend!
k

Ketan (kumare3)

04/01/2023, 1:28 AM
Thank you, ya kids time now
n

Nan Qin

04/03/2023, 4:55 PM
@Ketan (kumare3) when you get a chance, could you share docs/guides on how to change the default behavior (creating project-domain namespaces)?
k

Ketan (kumare3)

04/03/2023, 5:16 PM
Cc @Yee
y

Yee

04/03/2023, 5:16 PM
is this what you’re looking for @Nan Qin
n

Nan Qin

04/03/2023, 5:25 PM
I believe so
I used flyte-binary helm chart to deploy, do I just need to update this cm?
Copy code
apiVersion: v1
data:
  000-core.yaml: |
    admin:
      endpoint: localhost:8089
      insecure: true
    catalog-cache:
      endpoint: localhost:8081
      insecure: true
      type: datacatalog
    cluster_resources:
      standaloneDeployment: false
      templatePath: /etc/flyte/cluster-resource-templates
    logger:
      show-source: true
      level: 1
    propeller:
      create-flyteworkflow-crd: true
    webhook:
      certDir: /var/run/flyte/certs
      localCert: true
      secretName: flyte-backend-flyte-binary-webhook-secret
      serviceName: flyte-backend-flyte-binary-webhook
      servicePort: 443
  001-plugins.yaml: |
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - K8S-ARRAY
        default-for-task-types:
          - container: container
          - container_array: K8S-ARRAY
    plugins:
      logs:
        kubernetes-enabled: false
        cloudwatch-enabled: false
        stackdriver-enabled: false
      k8s-array:
        logs:
          config:
            kubernetes-enabled: false
            cloudwatch-enabled: false
            stackdriver-enabled: false
  002-database.yaml: |
    database:
      postgres:
        username: postgres
        passwordPath: /var/run/secrets/flyte/db-pass
        host: <RDS_HOST_DNS>
        port: 5432
        dbname: app
        options: "sslmode=disable"
  003-storage.yaml: |
    propeller:
      rawoutput-prefix: s3://<USER_DATA_BUCKET_NAME>/data
    storage:
      type: stow
      stow:
        kind: s3
        config:
          region: <AWS_REGION>
          disable_ssl: false
          v2_signing: false
          auth_type: iam
      container: <BUCKET_NAME>
  010-inline-config.yaml: |
    plugins:
      k8s:
        default-env-vars:
        - AWS_METADATA_SERVICE_TIMEOUT: 5
        - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
        inject-finalizer: true
    storage:
      cache:
        max_size_mbs: 100
        target_gc_percent: 100
    task_resources:
      defaults:
        cpu: 200m
        memory: 1Gi
        storage: 1Gi
      limits:
        cpu: 32
        memory: 128Gi
        storage: 512Gi
kind: ConfigMap
metadata:
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte-backend
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.16.0
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.4.3
  name: flyte-backend-flyte-binary-config
  namespace: flyte
or is it not possible with the flyte-binary chart? @Yee
y

Yee

04/03/2023, 9:17 PM
it is possible.
put it under the inline section?
just at the top level under the inline section should do it.
n

Nan Qin

04/03/2023, 9:21 PM
like this?
Copy code
010-inline-config.yaml: |
    namespace_mapping:
      template: custom_namespace 
    plugins:
      k8s:
        default-env-vars:
        - AWS_METADATA_SERVICE_TIMEOUT: 5
        - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
        inject-finalizer: true
    storage:
      cache:
        max_size_mbs: 100
        target_gc_percent: 100
    task_resources:
      defaults:
        cpu: 200m
        memory: 1Gi
        storage: 1Gi
      limits:
        cpu: 32
        memory: 128Gi
        storage: 512Gi
kind: ConfigMap
y

Yee

04/03/2023, 9:22 PM
yeah…
but you’re changing the helm chart right?
not like k edit cm
n

Nan Qin

04/03/2023, 9:22 PM
for all the tasks running in a namespace called 
custom_namespace
?
yeah I am patching with kustomize
y

Yee

04/03/2023, 9:23 PM
okay
well changing in helm is best
otherwise the next time you apply the helm chart the changes go away
what do you mean?
for all the tasks running in a namespace called 
custom_namespace
?
n

Nan Qin

04/03/2023, 9:25 PM
I mean if I want to make all the tasks running in a k8 namespace called 
custom_namespace
, no matter what flyte projects or domains these tasks belong. is it the right change?
Copy code
010-inline-config.yaml: |
    namespace_mapping:
      template: custom_namespace 

kind: ConfigMap
y

Yee

04/03/2023, 9:27 PM
yes that is correct
n

Nan Qin

04/03/2023, 9:28 PM
when you say changing in helm, do you mean adding it to 
values.yaml
for helm install to use?
another question about using inline config: Can all the config options in the documentation be updated/overriden with inline-config.yaml?
y

Yee

04/03/2023, 11:36 PM
yes
just make sure you do it correctly - a merge happens in the backend. so if you overwrite a list, you need to re-specify all the elements of the list that you want.
lists are replaced, maps are merged.
n

Nan Qin

04/04/2023, 1:08 AM
this is super helpful gratitude thank you
k

Ketan (kumare3)

04/04/2023, 1:27 AM
Cc @David Espejo (he/him) can we capture this
7 Views
Powered by