• Rahul Mehta

    Rahul Mehta

    3 months ago
    One more question that may be a bit naive: if there is a workflow in a project
    test
    , domain
    development
    & a version
    v1
    , how would I "promote" that workflow to
    staging
    in the same project w/ the same version? With what I've seen in the flytekit API, it only seems possible to register/create a workflow based on what's in the source code (ie. the decorated workflow function). Is it possible to "copy"/promote a workflow to a different domain only w/ the API? I also definitely could be misunderstanding how the domains abstraction is supposed to be applied to enable promotion/a proper CI/CD approach to pipelines
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    @Rahul Mehta great question. Unfortunately today we do not support intra Flyte promotion. The promotion is supposedly handled outside, in a proper ci/cd flow, which manager other general Purpose infra, where you also manage service accounts etc. Example at lyft here https://docs.flyte.org/en/latest/deployment/ideal_flow.html#case-study-mlops-at-lyft
  • Rahul Mehta

    Rahul Mehta

    3 months ago
    Thanks for the examples -- it makes sense that anything that's on
    master
    /
    main
    would by definition not be a "development" workflow.
  • Each domain, may change the data-directory, associated roles or some meta attributes like labels, annotations Is this something that's configured in the flyte-admin configmap, or elsewhere?
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    no when you register all of these are inputs
  • so for example when you run pyflyte run or
    flytectl register
    all of these things are inputs
  • cc @Alekhya / @Samhita Alla better docs on all the paramters that can be configured at different levels
  • So there is a heirarchy for config, When you create a project domain, you can now configure defaults for a project/domain combination, things like • Service account + iam role • raw data output buckets (this is where data will be automatically stored) • max parallelism • couple others • default cpu/mem for tasks where it is not specified Then when you register you can override these things and add more, like • labels / annotations etc • on termination behavior • notifications and for runtime all of this can be overriden
  • next couple weeks, docs for all of this will improve. cc @Niels Bantilan / @Haytham Abuelfutuh we need to probably have a clean flow of how these overrides work
  • Rahul Mehta

    Rahul Mehta

    3 months ago
    Intuitively I'd think stuff at the domain/project level would be something that'd be configurable with CRDs/config maps, vs the overrides which make sense to exist in the python source/at runtime. I did notice that it's possible to execute a workflow in a different domain/project with flytekit, but it doesn't appear to be possible with LaunchPlans. If it was possible, I was thinking about writing an operator that diffs the version of the last successful run in staging vs prod & auto-promotes it. I'd assume all of the appropriate metadata exists to do that
    wf = sandbox_remote.fetch_launch_plan(
        project="flytetester",
        domain="development",
        name="flyte_tests.example_workflow.example",
        version="<commit_hash>",
        )
    
    
    sandbox_remote.execute(
        entity=wf,
        domain="staging",
        inputs={},
    )
  • Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    So all overrides are at the control plane level. The goal Is also to abstract k8s so that it does not make users go mad and easier to scale to multiple clusters