https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Andrew

01/16/2024, 5:38 PM
Is there a way to use conditionals or some other feature to end a workflow early based on a condition. For example, if I have a workflow that processes data and then trains a model, but have a parameter
preprocess_only
, for example, is there a way to end after those tasks, and skip the rest, other than putting a conditional around all tasks after processing?
Or, a way to run several tasks inside one conditional, returning all of the values from the tasks, something like:
Copy code
train_metrics, eval_metrics, deploy_endpoint = conditional("preprocess_only").if_(preprocess_only.is_false()).then(
    train()
    evaluate()
    deploy()
)
s

Samhita Alla

01/19/2024, 12:02 PM
how about you call a subworkflow within a conditional? that subworkflow can include the train, evaluate and deploy tasks. makes sense?
a

Andrew

01/19/2024, 2:04 PM
That does make sense. I’ll play around with that, I’m not familiar with how subworkflows show up in the UI
@Samhita Alla I think that will work well! Quick follow-up. Is there a way to rename the “branchNode” label, to know what’s inside without having to open each one?
s

Samhita Alla

01/20/2024, 11:57 AM
would you mind sharing your code here? redacted should do.
a

Andrew

01/22/2024, 2:44 PM
for sure. here’s those two task calls, with conditionals
Copy code
pipeline_status_id = (
        conditional("create_status")
        .if_(production_run.is_true())
        .then(
            create_pipeline_status(...)
        )
        .else_()
        .then(noop())
    )

    dataset_bucket_dir = (
        conditional("dataset_bucket_dir")
        .if_(dataset_bucket_dir == "")
        .then(
            get_dataset_bucket_dir(...)
        )
        .else_()
        .then(return_str(value=dataset_bucket_dir))
    )
And the tasks are just decorated with
@task(container_image=ImageSpec(...))
This is my workaround right now for our use of conditionals. There may be a better way to handle it, I’m not sure, but we just have several tasks that we only want to run conditionally, depending on which parts of the pipeline we want to run
s

Samhita Alla

01/22/2024, 3:43 PM
This is my workaround right now for our use of conditionals.
i think subworkflow is the best technique. regarding conditionals, i think the latest version shouldn't display "branchNode". which version of flyte are you using?
a

Andrew

01/22/2024, 3:44 PM
Sounds good. I have started working in some subworkflows to clean some things up. I’ll continue with that and see if I can get it all into a better state. And I’m on
1.10.2
right now
for
flytekit
s

Samhita Alla

01/22/2024, 3:46 PM
what about flyte?
a

Andrew

01/22/2024, 3:49 PM
My bad. Is it this? Same version? Or how do I check that one?
Image from iOS.jpg
s

Samhita Alla

01/22/2024, 3:54 PM
oh, should be the helm chart version you installed.
i think it's the same. is it flyte-core or flyte-binary helm chart?
a

Andrew

01/22/2024, 4:03 PM
I deployed using this: https://github.com/davidmirror-ops/davidmirror-flyte Which I think is flyte-binary? But I’m not positive. And it looks like it may be 1.10:
version: v0.1.10 # VERSION
Oh wait, no its under flyte-core looks like
The commit sha that I’m on for that repo is
db3132ac910ddb8c68a643990ddf10eafb6163d3
if that helps
s

Samhita Alla

01/22/2024, 4:16 PM
have you used flyte-the-hard-way guide?
if so, it's flyte-binary.
a

Andrew

01/22/2024, 4:21 PM
Shoot, I sent the wrong repo, its actually this one: https://github.com/unionai-oss/deploy-flyte, and I’m on GCP, so I used this guide: https://github.com/unionai-oss/deploy-flyte/blob/main/environments/gcp/flyte-core/README.md along with some back and forth with David because some things weren’t quite working yet when I first tried it
And that’s under a
flyte-core
directory, so that’s where I pulled that from. Sorry for the confusion
s

Samhita Alla

01/22/2024, 4:29 PM
sorry. looks like it doesn't have to do with the latest version. i'll get back to you.
a

Andrew

01/22/2024, 4:29 PM
all good, thanks!
Hey, just wanted to check back in and see if anything was discovered here. Thanks!
s

Samhita Alla

01/30/2024, 5:20 PM
hey, sorry i didn't get back to you earlier. this will be fixed in open source real soon!
a

Andrew

01/30/2024, 5:25 PM
Sounds great, thank you!
2 Views