https://flyte.org
Join Slack
I’m trying to execute a simple test workflow and getting this error: `Request failed with status cod...
# flyte-support
s
I’m trying to execute a simple test workflow and getting this error: 
Request failed with status code 500 failed to create workflow in propeller namespaces <project-name> not found
. The project already shows up in the flyte console and I was able to package/register the workflow, but I can’t seem to execute it?
f
Did you create the namespace that matches the project
s
No, is that required?
So you can’t just run 
POST /projects
? You need a matching namespace created?
f
Then you need the cluster resource manager
Or you need to change the namespaces to be constant
s
Is this documented somewhere? I thought this was it, but maybe I’m looking at the wrong place: https://docs.flyte.org/projects/cookbook/en/latest/auto/remote_access/register_project.html
f
The default helm And deployment templates have them Configured
s
OK I’ll review that thanks - is there anywhere in the docs where the project creation process is detailed?
f
Cc @acceptable-policeman-57188
s
So if you turn off the cluster resource manager, you can’t create a project? Or does it just default to no limits?
@freezing-airport-6809
f
currently the default config assumes it is enabled
but it is possible to turn it off and just use one namespace
will get back to you with those docs tomorrow
s
OK great thank you
Hopefully it explains how cluster resource manager, project and namespaces are related because I’m still a bit confused about how all three is related
And it looks like it can be updated after the cluster is created (via 
flytectl update cluster-resource-attribute
) but would like to confirm that as well
f
yes it can be
as long as you use cluster resource manager
s
OK that’s great
f
and thank you for the patience
as you probably have imagined
documenting flyte is like documenting k8s
👍 1
would appreciate help 😄
s
Yes I get it
👍 1
Once I get a better grip of this, would love to contribute at some point!
Thanks for your help with this
f
it may happen that some doc may not be present, but we will definitely try to add some context
👍 1
s
That’d be great - thanks
a
hey @sticky-angle-28419 here's some documentation on what the cluster resource controller does: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#cluster-resources including managing per-<project>-<domain> namespaces, these are the templates that currently get applied: https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L737,L760
if you want just one namespace or per-domain namespaces for your project, you can change the namespace mapping template in the flyteadmin config: https://docs.flyte.org/en/latest/deployment/cluster_config/scheduler_config.html#section-namespace-mapping
this latter option is useful if you don't have the cluster resource controller running and want to manually configure a pre-defined set of namespaces, like Ketan mentioned
s
OK thanks - are there any examples I can look at? Also, is it possible to use a single namespace for the project? If so, how can I configure that?
a
is it possible to use a single namespace for the project
yup, the default config has 
'{{ project }}-{{ domain }}'
but you can always set it to 
'{{ project }}'
s
And where are these values placed? In helm values?
Can I see any examples?
a
e.g.
Copy code
namespace:
  namespace_mapping:
    template: "{{ project }}"
s
Thanks - so to summarize then, to create a Flyte project with this setup, I can simply add this to values.yaml for helm and redeploy, create a project either via cli or rest api, and then create a namespace that matches the project name - with the cluster resource manager disabled?
a
yup, if you're okay manually maintaining the per-project namespaces that should work
s
Wait - there’s an automatic way to do it?
a
that's what cluster resource controller does 🙂
it automatically provisions per-project namespaces with things like resource quota using the templates i shared earlier
this way you can dynamically add new projects using the cli without any additional set-up on your side
s
ohhh I see now
So in the templates you shared, aa_namespace is the rule for creating name space and ab_project_resource_quota is the rule for creating resource quota and they’re create dynamically based on project creation?
Does this apply to using rest api as well or only cli?
a
both, the cluster resource controller reads from the db where the projects ultimately reside
s
OK got it - I’ll try this. Thank you!
👍 1
@acceptable-policeman-57188 I was redeploying helm chart and it keeps failing - this is the error message: 
<https://2A6739B7813451087E3258C60BC37CF4.yl4.us-west-1.eks.amazonaws.com/api/v1/namespaces/flyte/services/flyte-release-kubernetes-dashboard>": context deadline exceeded
Any idea what is causing this? I was able to deploy fine before (although I did add cluster resource manager to values now but it doesn’t seem related? Also, the cluster resource manager seems to be working - it created namespaces for my project)
f
It seems it's failing to deploy k8s dashboard
s
Hmm I think it might be something else since it gives me different errors every time I try. This time I got this: 
failed to become available within allocated timeout. Error: Helm Release flyte/flyte-release: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
OK so it was the cluster resource manager setting - I must have gotten something wrong there because disabling deploys fine. Will report back after trying a couple things
So I figured out what the issue is: making the cluster_resource_manager quotas really high makes helm release fails. I used the number in here: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html which is 1,000 cpus and 5TB of memory. Is there a limit to the quota we can set?
f
At many companies we have set high numbers
It should not affect
Maybe typo
s
Yeah it must have been - just tried with 1000 cpus and 5Ti in memory and it worked - thanks!
g
Hi, I have troubled with the same issue of @sticky-angle-28419, but the 
project - domain
mapping cannot be solved. Would you give me an advice? (flyte.yaml file is values.yaml below)
I think this format has been changed into this.
Copy code
namespace_config:
  namespace_mapping:
    template: "{{ project }}"
only 
namespace
does not mapping project into one domain.
a
the namespace mapping is only used for the kubernetes namespace used to launch flyte workflow execution pods
the console will still show separate domains - if you want just one domain you can modify the flyteadmin config
note that domains are a required construct in flyte deployments - but you can always just one global domain
240 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by