Dennis O'Brien
09/22/2023, 12:08 AMpyflyte backfill
and I messed something up with the parallelism. Now the workflow is in the state FAILING
and I don't see a way to cancel this or clean it up. If I try to run another backfill with (hopefully) fixed parallelism settings, I get an error:
RPC Failed, with Status: StatusCode.INVALID_ARGUMENT
details: workflow with different structure already exists
Debug string UNKNOWN:Error received from peer {grpc_message:"workflow with different structure already exists", grpc_status:3, created_time:"2023-09-21T16:42:09.869845-07:00"}
Is there a way to clear/cancel the backfill via command line or the Flyte UI?max_parallelism=2
and passed --parallel
to pyflyte backfill
. Instead of launching two executions at a time, it started 25. Things started failing at the stage of writing to the database (something I need to fix) and the backfill workflow has been in the state FAILING
for the last hour.
The command I am trying to run now is
pyflyte --config path/to/config.yml backfill \
--project my-project-name \
--domain prod \
--from-date 2023-01-25 \
--to-date 2023-03-25 \
--serial \
launch_plan_name
Samhita Alla
Ketan (kumare3)
Dennis O'Brien
11/10/2023, 7:14 PMYou should be able to cancel the workflow, right?I don't see a way to cancel the previous backfill workflow. When I browse to this workflow in the Flyte UI, the only action I can take is "Launch Workflow". I tried using
flytectl get execution
to list the existing execution so I could then try flytectl delete execution
but I'm not familiar enough with flytectl
to make any progress. I'm not even sure it's a viable route.
Hmm do you need a max parallel controlI assumed the using
--parallel
would use the max_parallelism
from the launch plan being referenced. But it doesn't seem to do that. So I think either 1) use the existing max_parallelism
from the launch plan, or 2) support something like --parallelism=2
.Ketan (kumare3)
Dennis O'Brien
11/14/2023, 1:59 AMKetan (kumare3)
Dennis O'Brien
11/16/2023, 6:34 PM--execution-name
thinking that might allow new backfill with a different name, but I get the same error.
pyflyte --config marketplace/pa/dags/bpo_ltv_model/flyte/config.yml backfill \
--project marketplace--pa--dags--bpo-ltv-model \
--domain dprod \
--from-date 2022-10-01 \
--to-date 2023-01-01 \
--serial \
--execution-name backfill-bpo_ltv_180d_horizon_1d_observation_20221001_20230101 \
bpo_ltv_180d_horizon_1d_observation
...
RPC Failed, with Status: StatusCode.INVALID_ARGUMENT
details: workflow with different structure already exists
Debug string UNKNOWN:Error received from peer {grpc_message:"workflow with different structure already exists", grpc_status:3, created_time:"2023-11-16T10:33:05.190869-08:00"}
Let me know if this is better handled as a github issue. I'm happy to create one if that is best. I'm just assuming I'm going about something wrong.Samhita Alla
Dennis O'Brien
11/17/2023, 11:49 PMVersion for the registered workflow. If not specified it is auto-derived using the start and end dateIf I look at the versions available for one of my workflows, they are the available tags for the docker image with the default being the latest version. Is this the same meaning for "version"?
Samhita Alla
Dennis O'Brien
11/20/2023, 6:10 PM