seunggs

    seunggs

    1 week ago
    I’m trying to execute a simple test workflow and getting this error:
    Request failed with status code 500 failed to create workflow in propeller namespaces <project-name> not found
    . The project already shows up in the flyte console and I was able to package/register the workflow, but I can’t seem to execute it?
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    Did you create the namespace that matches the project
    seunggs

    seunggs

    1 week ago
    No, is that required?
    So you can’t just run
    POST /projects
    ? You need a matching namespace created?
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    Then you need the cluster resource manager
    Or you need to change the namespaces to be constant
    seunggs

    seunggs

    1 week ago
    Is this documented somewhere? I thought this was it, but maybe I’m looking at the wrong place: https://docs.flyte.org/projects/cookbook/en/latest/auto/remote_access/register_project.html
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    The default helm And deployment templates have them Configured
    seunggs

    seunggs

    1 week ago
    OK I’ll review that thanks - is there anywhere in the docs where the project creation process is detailed?
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    Cc @katrina
    seunggs

    seunggs

    1 week ago
    So if you turn off the cluster resource manager, you can’t create a project? Or does it just default to no limits?
    @Ketan (kumare3)
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    currently the default config assumes it is enabled
    but it is possible to turn it off and just use one namespace
    will get back to you with those docs tomorrow
    seunggs

    seunggs

    1 week ago
    OK great thank you
    Hopefully it explains how cluster resource manager, project and namespaces are related because I’m still a bit confused about how all three is related
    And it looks like it can be updated after the cluster is created (via
    flytectl update cluster-resource-attribute
    ) but would like to confirm that as well
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    yes it can be
    as long as you use cluster resource manager
    seunggs

    seunggs

    1 week ago
    OK that’s great
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    and thank you for the patience
    as you probably have imagined
    documenting flyte is like documenting k8s
    would appreciate help 😄
    seunggs

    seunggs

    1 week ago
    Yes I get it
    Once I get a better grip of this, would love to contribute at some point!
    Thanks for your help with this
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    it may happen that some doc may not be present, but we will definitely try to add some context
    seunggs

    seunggs

    1 week ago
    That’d be great - thanks
    k

    katrina

    1 week ago
    hey @seunggs here's some documentation on what the cluster resource controller does: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#cluster-resources including managing per-<project>-<domain> namespaces, these are the templates that currently get applied: https://github.com/flyteorg/flyte/blob/1e3d515550cb338c2edb3919d79c6fa1f0da5a19/charts/flyte-core/values.yaml#L737,L760
    if you want just one namespace or per-domain namespaces for your project, you can change the namespace mapping template in the flyteadmin config: https://docs.flyte.org/en/latest/deployment/cluster_config/scheduler_config.html#section-namespace-mapping
    this latter option is useful if you don't have the cluster resource controller running and want to manually configure a pre-defined set of namespaces, like Ketan mentioned
    seunggs

    seunggs

    1 week ago
    OK thanks - are there any examples I can look at? Also, is it possible to use a single namespace for the project? If so, how can I configure that?
    k

    katrina

    1 week ago
    is it possible to use a single namespace for the project
    yup, the default config has
    '{{ project }}-{{ domain }}'
    but you can always set it to
    '{{ project }}'
    seunggs

    seunggs

    1 week ago
    And where are these values placed? In helm values?
    Can I see any examples?
    k

    katrina

    1 week ago
    e.g.
    namespace:
      namespace_mapping:
        template: "{{ project }}"
    seunggs

    seunggs

    1 week ago
    Thanks - so to summarize then, to create a Flyte project with this setup, I can simply add this to values.yaml for helm and redeploy, create a project either via cli or rest api, and then create a namespace that matches the project name - with the cluster resource manager disabled?
    k

    katrina

    1 week ago
    yup, if you're okay manually maintaining the per-project namespaces that should work
    seunggs

    seunggs

    1 week ago
    Wait - there’s an automatic way to do it?
    k

    katrina

    1 week ago
    that's what cluster resource controller does 🙂
    it automatically provisions per-project namespaces with things like resource quota using the templates i shared earlier
    this way you can dynamically add new projects using the cli without any additional set-up on your side
    seunggs

    seunggs

    1 week ago
    ohhh I see now
    So in the templates you shared, aa_namespace is the rule for creating name space and ab_project_resource_quota is the rule for creating resource quota and they’re create dynamically based on project creation?
    Does this apply to using rest api as well or only cli?
    k

    katrina

    1 week ago
    both, the cluster resource controller reads from the db where the projects ultimately reside
    seunggs

    seunggs

    1 week ago
    OK got it - I’ll try this. Thank you!
    @katrina I was redeploying helm chart and it keeps failing - this is the error message:
    <https://2A6739B7813451087E3258C60BC37CF4.yl4.us-west-1.eks.amazonaws.com/api/v1/namespaces/flyte/services/flyte-release-kubernetes-dashboard>": context deadline exceeded
    Any idea what is causing this? I was able to deploy fine before (although I did add cluster resource manager to values now but it doesn’t seem related? Also, the cluster resource manager seems to be working - it created namespaces for my project)
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    It seems it's failing to deploy k8s dashboard
    seunggs

    seunggs

    1 week ago
    Hmm I think it might be something else since it gives me different errors every time I try. This time I got this:
    failed to become available within allocated timeout. Error: Helm Release flyte/flyte-release: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    OK so it was the cluster resource manager setting - I must have gotten something wrong there because disabling deploys fine. Will report back after trying a couple things
    So I figured out what the issue is: making the cluster_resource_manager quotas really high makes helm release fails. I used the number in here: https://docs.flyte.org/en/latest/deployment/cluster_config/general.html which is 1,000 cpus and 5TB of memory. Is there a limit to the quota we can set?
    Ketan (kumare3)

    Ketan (kumare3)

    1 week ago
    At many companies we have set high numbers
    It should not affect
    Maybe typo
    seunggs

    seunggs

    1 week ago
    Yeah it must have been - just tried with 1000 cpus and 5Ti in memory and it worked - thanks!