Hello Everyone! I am trying to execute a launchplan with pyflyte and I have encountered two issues: 1. I am able to execute a launchplan even if it is disabled. I am not sure if this is the expected behavior. If this is expected, then how do I pause the pipeline? 2. When I set a launchplan to an older version (like a rollback), the workflow executes with the latest version and not the active version. Both of these issues seem like bugs, either in the Flyte server or in pyflyte. Can someone clarify the expected behaviors?

what do you mean execute?

$ pyflyte run remote-launchplan <lp_name>

I don't see the option to pick a version in pyflyte. If I set the (X - 1) version active from UI, my mental model expect to trigger (X-1) version and not X.

ohh yes we should add a version

Thanks @freezing-airport-6809! I see similar issue in flytectl as well. execution_spec pick the latest version and not the active version.

flytectl get launchplan  --project flytesnacks --domain development  <lp_name>  --execFile execution_spec.yaml

The only way to fix this is by looking at all LPs, find the ACTIVE version, then get execution_spec and then execute the LP. I am not sure if there is better way.

./bin/flytectl get launchplan  --project flytesnacks --domain development  <lp_name>  -o yaml | grep ACTIVE

It would be great if you could consider feature request to pause the pipeline. Here's my use case: someone pushed a bad change to production. For some reason, I can't roll back to the old version due to some internal data issue. In Airflow, I can just pause the DAG, put out the fix and enable again. Flyte registers the default launch plan. I'm comparing the default launch plan with Airflow DAG. Flyte has added a version control feature, which is excellent and missing in airflow, we still need a way to pause a pipeline. This could be done by either rejecting incoming requests or adding the executions to a queued state.

@worried-iron-1001 i thought some more you are actually right, we should use active. I think it makes more sense. Let me create that PR and then we will add a version PR too? what say?

Yes. Let me do it tomorrow morning. Regarding - is this a breaking behavior? I don't think so. When we activate a launchplan, we have to specify a version. That is my contract with the system to activate a specific version. If there is a new version, I can test with workflow and I can decide if I want to activate or not. Maybe flag like --activate-latest or something similar might work.

pyflyte launchplan  --activate/--deactivate <lp_name> <version>

Btw, I really like like your Flyte UI/UX (apart from many other things). It is lighting fast. ⚡

ohh highly appreciate the feedback. I would highly recommend to try out Union. Its even faster 😄

Thank you for fast fix: Here is the issue ticket: https://github.com/flyteorg/flyte/issues/5834

Thank you for sharing @worried-iron-1001! The idea of "pausing" an execution makes a ton of sense. I haven't seen any other requests for this functionality so far, and it would only be applicable for problems where you have some "external state" (IE: a database),

You can infact pause - it’s like deactivate and then activate. Do you mean you want to pause and restart where it should backfill?

Current workflow activate andd deactivate works incontext of cronjobs or schedules. I have seen uses cases (including our) where there is external trigger like Kafka, Service, Lambda, or manual triggering. I know these external triggers are not supported in Flyte today, except SQS. We use flyte go gRPC client to execute the remote launchplans.

Hi Ashwin! Yes, external triggers (Kafka, API, Webhooks, Lambda, Kubectl), etc are things we are actively exploring.

@astonishing-eve-54331: Instead of adding N external triggers in flyte, I think it is better to add a feature to pause the LP scheduler if is deactivated. Example: 1. Let's say I have LP which is currently activated. 2. Execution A, B, C are submitted to LP and executed. 3. For some reason, we need to deactivate the LP 4. User submitted P, Q, R request, but requests are not executed since LP is deactivated. 5. A fix is landed and LP version updated and activated. 6. Now P, Q, R will run with the activated version, unless specified otherwise.

i dont think we will add triggers to flyte, as you have an api and anyone can build triggers - well in union that is part of the platform