#4020 [BUG] Flytepropeller bug for multicluster operation
Issue created by
samuel-sujith
Describe the bug
I am running Flyte in multicluster mode and I am getting the below error intermittently.
I followed the instructions here
https://docs.flyte.org/en/latest/deployment/deployment/multicluster.html
Operation cannot be fulfilled on
flyteworkflows.flyte.lyft.com "a52mgxmqsx682x9kkbzr": the object has been modified; please apply your changes to the latest version and try again
When i submit the job and it goes to the remote cluster, after one or 2 nodes are executed, I get this error in the remote data plane cluster flytepropeller. I think this means that the original workflow has been edited and since that version doesnt exist in the remote, it errors out and the node execution aborts.
The error is coming from flytepropeller workflowstore/passthrough.go in the line number 104
logger.Errorf(ctx, "Failed to update workflow. Error [%v]", err)
But if i relaunch the execution, it generally goes fine without any issues.
Is this a known issue with multicluster operation
Expected behavior
When i execute multicluster operation, the workflow should not randomly fail
Additional context to reproduce
https://docs.flyte.org/en/latest/deployment/deployment/multicluster.html
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyte