https://flyte.org logo
Join Slack
Powered by
Hi! I'm working on helping our ML team get Flyte r...
# flyte-support
n
Hi! I'm working on helping our ML team get Flyte running in our production environments - but I've run into https://github.com/flyteorg/flyte/issues/3827. I have opened a more detailed https://github.com/flyteorg/flyte/issues/6533 issue. I'm concerned though about. whether or not the Flyte development team would accept PRs that help make the 
flyte-core
chart installable in a namespace-scoped way. Before we try to do that work, is this something that the Flyte team feels like they would support?
cc @cool-nest-98527
👀 2
c
I'm not not from Union but my impression is that there are probably cluster roles because by default flyteworkflows have namespaces set to `{project}-{domain}. So flyteworkflows naturally are placed all over the cluster. Making the resource more tightly scoped would be possible, it may be a little brittle but if its all backwards compatible I wouldn't mint reviewing it.
n
So that is another thing we haven't totally figured out - we have disabled the 
cluster_resource_manager
because we cannot allow Flyte to manage namespaces. Our expectation is that we can configure Flyte to launch resources in its own namespace only - rather than using unique namespaces for each project. I hadn't seen the 
namespaces
key though in there... but I am hoping that by disabling the resource manager, we just bypass that whole system.
c
We also don't use the 
cluster_resource_manager
. I think that is for further customizing the workflow resources.
But even without the 
cluster_resource_manager
you'll stil need to configure the namespace mapping configuration as that is what is used by 
flyteadmin
to generate the source resource (before the cluster resource manager would theoretically modify it)
🙏 1
n
Yeah - we haven't gotten to that part of the configuration but I am hoping we'll be able to configure it to just use the same namespace that Flyte runs in. Do you foresee that being a problem?
c
Not off the top of my head
c
a few ideas: • propeller has a limit-namespace setting that restricts its operation to a single Kubernetes namespace. This means the Flyte Propeller instance will only process workflows and related resources within the specified namespace. It can be set to 
all
(default) or a single other namespace. I’m not sure if this means the control plane would be installed in the same namespace as where tasks run, but we could try it. I’ve read that the service accounts used by Flyte will only require a 
Role
and 
RoleBinding
, not their cluster-wide equivalents in this configuration. • propeller manager can be used with the include / exclude flyte project / domain configuration values on propeller. These were originally introduced for using propeller manager to automatically scale-out propeller. You could use this component, or if you wanted to manually manage each propeller instance (rather than having the manager handle them all) you can pass specific values for inlcude / exclude. (slack)
@nutritious-piano-11388 i may have gotten it to work with these settings for flyte-core section of values.yaml:
Copy code
cluster_resource_manager:
    enabled: false

  configmap:
    namespace_config:
      namespace_mapping:
        template: '{{ .Release.Namespace }}'
    core:
      propeller:
        limit-namespace: '{{ .Release.Namespace }}'
    domain:
      domains:
        - id: development
          name: development
          namespace: '{{ .Release.Namespace }}'
        - id: staging
          name: staging
          namespace: '{{ .Release.Namespace }}'
        - id: production
          name: production
          namespace: '{{ .Release.Namespace }}'
attached is a screenshot of a task pod 
ajx4…
running in the flyte namespace alongside the control plane. and here’s my namespaces:
Copy code
$ k get namespaces | grep flyte
flyte                Active   5m21s
i think it still tries to install the CRD, ClusterRole and ClusterRoleBinding, though…
👍 1
n
Excellent, thanks @cool-nest-98527... We still need to solve some of the other challenges, but I may focus my efforts tomorrow on putting up a PR to optionalize the Cluster-vs-Namespace scoped mode.
🙏 1
I've put up https://github.com/flyteorg/flyte/pull/6534/files as an initial proposal.
🙏 1
✅ 1
a
thank you all 🙇🏽
🙏 1
n
@average-finland-92144 Hey... I re-ran 
make helm
but it didn't fix the 
redis: null
thing... and each time I run it, we get a different randomly generated password (which is a behavior in helm charts that I can't stand). Would you try checking it out and seeing if you get a different 
make helm
output?
👀 1
a
I get the same output (same hash) you shared from running 
make helm
in your fork
Copy code
diff --git a/deployment/sandbox/flyte_helm_generated.yaml b/deployment/sandbox/flyte_helm_generated.yaml
index 1edf46e6c..7edd98f9f 100644
--- a/deployment/sandbox/flyte_helm_generated.yaml
+++ b/deployment/sandbox/flyte_helm_generated.yaml
@@ -671,7 +671,6 @@ data:
   resource_manager.yaml: |
     propeller:
       resourcemanager:
-        redis: null
         type: noop
   storage.yaml: |
     storage:
@@ -7185,7 +7184,7 @@ spec:
   template:
     metadata:
       annotations:
-        configChecksum: "580cd917ae7b7ed52d3100047b252b68a265175f2572a5803417473bb9e5e8d"
+        configChecksum: "fe60472a10d9e71dea69ddfa28ab9232b15b80cb3453af79354a885e1b6a93f"
@nutritious-piano-11388 ^
n
So my guess is that its because I have a newer version of helm, and helm has changd how it handles null values recently..
Copy code
version.BuildInfo{Version:"v3.18.2", GitCommit:"04cad4610054e5d546aa5c5d9c1b1d5cf68ec1f8", GitTreeState:"clean", GoVersion:"go1.24.3"}
Currently the 
script/generate_helm.sh
script installs a specific (
v3.15.4
) version of helm, but .. if its already installed, it doesn't properly override it with an older version:
Copy code
% make helm
fatal: No names found, cannot describe anything.
Makefile:130: warning: overriding commands for target `go-tidy'
boilerplate/flyte/golang_test_targets/Makefile:59: warning: ignoring old commands for target `go-tidy'
bash script/generate_helm.sh
+ echo 'Generating Helm'
Generating Helm
+ HELM_SKIP_INSTALL=false
+ '[' false '!=' true ']'
+ curl <https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3>
+ DESIRED_VERSION=v3.15.4
+ bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11913  100 11913    0     0  75355      0 --:--:-- --:--:-- --:--:-- 75398
Helm v3.15.4 is already v3.15.4
+ helm version
version.BuildInfo{Version:"v3.18.2", GitCommit:"04cad4610054e5d546aa5c5d9c1b1d5cf68ec1f8", GitTreeState:"clean", GoVersion:"go1.24.3"}
So ... what should we do? Should I just bump the version of helm in the generate_helm script so its more modern?
(or, the script could be updated so it installs helm into a local bin directory inside the repo... and then explicitly calls that helm?)
3 Views
Open in Slack
PreviousNext
Powered by