https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
r

Rahul Mehta

04/24/2022, 11:00 PM
One more question that may be a bit naive: if there is a workflow in a project 
test
, domain 
development
& a version 
v1
, how would I "promote" that workflow to 
staging
in the same project w/ the same version? With what I've seen in the flytekit API, it only seems possible to register/create a workflow based on what's in the source code (ie. the decorated workflow function). Is it possible to "copy"/promote a workflow to a different domain only w/ the API? I also definitely could be misunderstanding how the domains abstraction is supposed to be applied to enable promotion/a proper CI/CD approach to pipelines
k

Ketan (kumare3)

04/24/2022, 11:57 PM
@Rahul Mehta great question. Unfortunately today we do not support intra Flyte promotion. The promotion is supposedly handled outside, in a proper ci/cd flow, which manager other general Purpose infra, where you also manage service accounts etc. Example at lyft here https://docs.flyte.org/en/latest/deployment/ideal_flow.html#case-study-mlops-at-lyft
r

Rahul Mehta

04/25/2022, 12:03 AM
Thanks for the examples -- it makes sense that anything that's on `master`/`main` would by definition not be a "development" workflow.
Each domain, may change the data-directory, associated roles or some meta attributes like labels, annotations
Is this something that's configured in the flyte-admin configmap, or elsewhere?
k

Ketan (kumare3)

04/25/2022, 12:23 AM
no when you register all of these are inputs
so for example when you run pyflyte run or 
flytectl register
all of these things are inputs
👍 1
cc @Alekhya / @Samhita Alla better docs on all the paramters that can be configured at different levels
👍 2
So there is a heirarchy for config, When you create a project domain, you can now configure defaults for a project/domain combination, things like • Service account + iam role • raw data output buckets (this is where data will be automatically stored) • max parallelism • couple others • default cpu/mem for tasks where it is not specified Then when you register you can override these things and add more, like • labels / annotations etc • on termination behavior • notifications and for runtime all of this can be overriden
next couple weeks, docs for all of this will improve. cc @Niels Bantilan / @Haytham Abuelfutuh we need to probably have a clean flow of how these overrides work
👍 1
r

Rahul Mehta

04/25/2022, 1:02 AM
Intuitively I'd think stuff at the domain/project level would be something that'd be configurable with CRDs/config maps, vs the overrides which make sense to exist in the python source/at runtime. I did notice that it's possible to execute a workflow in a different domain/project with flytekit, but it doesn't appear to be possible with LaunchPlans. If it was possible, I was thinking about writing an operator that diffs the version of the last successful run in staging vs prod & auto-promotes it. I'd assume all of the appropriate metadata exists to do that
Copy code
wf = sandbox_remote.fetch_launch_plan(
    project="flytetester",
    domain="development",
    name="flyte_tests.example_workflow.example",
    version="<commit_hash>",
    )


sandbox_remote.execute(
    entity=wf,
    domain="staging",
    inputs={},
)
k

Ketan (kumare3)

04/25/2022, 1:13 AM
So all overrides are at the control plane level. The goal Is also to abstract k8s so that it does not make users go mad and easier to scale to multiple clusters
2 Views
Powered by