https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

seunggs

09/12/2022, 3:00 AM
I’m trying to execute a simple test workflow and getting this error: 
Request failed with status code 500 failed to create workflow in propeller namespaces <project-name> not found
. The project already shows up in the flyte console and I was able to package/register the workflow, but I can’t seem to execute it?
k

Ketan (kumare3)

09/12/2022, 3:27 AM
Did you create the namespace that matches the project
s

seunggs

09/12/2022, 3:28 AM
No, is that required?
So you can’t just run 
POST /projects
? You need a matching namespace created?
k

Ketan (kumare3)

09/12/2022, 3:36 AM
Then you need the cluster resource manager
Or you need to change the namespaces to be constant
s

seunggs

09/12/2022, 3:37 AM
Is this documented somewhere? I thought this was it, but maybe I’m looking at the wrong place: https://docs.flyte.org/projects/cookbook/en/latest/auto/remote_access/register_project.html
k

Ketan (kumare3)

09/12/2022, 3:39 AM
The default helm And deployment templates have them Configured
s

seunggs

09/12/2022, 3:41 AM
OK I’ll review that thanks - is there anywhere in the docs where the project creation process is detailed?
k

Ketan (kumare3)

09/12/2022, 3:41 AM
Cc @katrina
s

seunggs

09/12/2022, 3:49 AM
So if you turn off the cluster resource manager, you can’t create a project? Or does it just default to no limits?
@Ketan (kumare3)
k

Ketan (kumare3)

09/12/2022, 3:50 AM
currently the default config assumes it is enabled
but it is possible to turn it off and just use one namespace
will get back to you with those docs tomorrow
s

seunggs

09/12/2022, 3:50 AM
OK great thank you
Hopefully it explains how cluster resource manager, project and namespaces are related because I’m still a bit confused about how all three is related
And it looks like it can be updated after the cluster is created (via 
flytectl update cluster-resource-attribute
) but would like to confirm that as well
k

Ketan (kumare3)

09/12/2022, 3:53 AM
yes it can be
as long as you use cluster resource manager
s

seunggs

09/12/2022, 3:54 AM
OK that’s great
k

Ketan (kumare3)

09/12/2022, 3:54 AM
and thank you for the patience
as you probably have imagined
documenting flyte is like documenting k8s
would appreciate help 😄
s

seunggs

09/12/2022, 3:54 AM
Yes I get it
Once I get a better grip of this, would love to contribute at some point!
Thanks for your help with this
k

Ketan (kumare3)

09/12/2022, 4:02 AM
it may happen that some doc may not be present, but we will definitely try to add some context
s

seunggs

09/12/2022, 4:02 AM
That’d be great - thanks
k

katrina

09/12/2022, 3:51 PM
hey @seunggs here's some documentation on what the cluster resource controller does: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#cluster-resources including managing per-<project>-<domain> namespaces, these are the templates that currently get applied: https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L737,L760
if you want just one namespace or per-domain namespaces for your project, you can change the namespace mapping template in the flyteadmin config: https://docs.flyte.org/en/latest/deployment/cluster_config/scheduler_config.html#section-namespace-mapping
this latter option is useful if you don't have the cluster resource controller running and want to manually configure a pre-defined set of namespaces, like Ketan mentioned
s

seunggs

09/12/2022, 3:53 PM
OK thanks - are there any examples I can look at? Also, is it possible to use a single namespace for the project? If so, how can I configure that?
k

katrina

09/12/2022, 3:54 PM
is it possible to use a single namespace for the project
yup, the default config has 
'{{ project }}-{{ domain }}'
but you can always set it to 
'{{ project }}'
s

seunggs

09/12/2022, 3:56 PM
And where are these values placed? In helm values?
Can I see any examples?
k

katrina

09/12/2022, 3:57 PM
e.g.
Copy code
namespace:
  namespace_mapping:
    template: "{{ project }}"
s

seunggs

09/12/2022, 4:01 PM
Thanks - so to summarize then, to create a Flyte project with this setup, I can simply add this to values.yaml for helm and redeploy, create a project either via cli or rest api, and then create a namespace that matches the project name - with the cluster resource manager disabled?
k

katrina

09/12/2022, 4:34 PM
yup, if you're okay manually maintaining the per-project namespaces that should work
s

seunggs

09/12/2022, 4:34 PM
Wait - there’s an automatic way to do it?
k

katrina

09/12/2022, 4:34 PM
that's what cluster resource controller does 🙂
it automatically provisions per-project namespaces with things like resource quota using the templates i shared earlier
this way you can dynamically add new projects using the cli without any additional set-up on your side
s

seunggs

09/12/2022, 4:35 PM
ohhh I see now
So in the templates you shared, aa_namespace is the rule for creating name space and ab_project_resource_quota is the rule for creating resource quota and they’re create dynamically based on project creation?
Does this apply to using rest api as well or only cli?
k

katrina

09/12/2022, 4:37 PM
both, the cluster resource controller reads from the db where the projects ultimately reside
s

seunggs

09/12/2022, 4:38 PM
OK got it - I’ll try this. Thank you!
@katrina I was redeploying helm chart and it keeps failing - this is the error message: 
<https://2A6739B7813451087E3258C60BC37CF4.yl4.us-west-1.eks.amazonaws.com/api/v1/namespaces/flyte/services/flyte-release-kubernetes-dashboard>": context deadline exceeded
Any idea what is causing this? I was able to deploy fine before (although I did add cluster resource manager to values now but it doesn’t seem related? Also, the cluster resource manager seems to be working - it created namespaces for my project)
k

Ketan (kumare3)

09/13/2022, 3:09 AM
It seems it's failing to deploy k8s dashboard
s

seunggs

09/13/2022, 4:35 AM
Hmm I think it might be something else since it gives me different errors every time I try. This time I got this: 
failed to become available within allocated timeout. Error: Helm Release flyte/flyte-release: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
OK so it was the cluster resource manager setting - I must have gotten something wrong there because disabling deploys fine. Will report back after trying a couple things
So I figured out what the issue is: making the cluster_resource_manager quotas really high makes helm release fails. I used the number in here: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html which is 1,000 cpus and 5TB of memory. Is there a limit to the quota we can set?
k

Ketan (kumare3)

09/13/2022, 1:41 PM
At many companies we have set high numbers
It should not affect
Maybe typo
s

seunggs

09/13/2022, 3:31 PM
Yeah it must have been - just tried with 1000 cpus and 5Ti in memory and it worked - thanks!
s

SeungTaeKim

10/24/2022, 2:20 AM
Hi, I have troubled with the same issue of @seunggs, but the 
project - domain
mapping cannot be solved. Would you give me an advice? (flyte.yaml file is values.yaml below)
I think this format has been changed into this.
Copy code
namespace_config:
  namespace_mapping:
    template: "{{ project }}"
only 
namespace
does not mapping project into one domain.
k

katrina

10/24/2022, 4:06 PM
the namespace mapping is only used for the kubernetes namespace used to launch flyte workflow execution pods
the console will still show separate domains - if you want just one domain you can modify the flyteadmin config
note that domains are a required construct in flyte deployments - but you can always just one global domain
22 Views
Powered by