Hey, I am trying to run a flytesnacks workflow on Flyte deployed on our self-managed Kubernetes cluster (v1.18.2). The workflow executed by

pyflyte run --remote

is in the

Running

state in the console, a

workflow

is created but in propeller there is an error logged:

E1117 09:51:41.531924       1 workers.go:102] error syncing 'flytesnacks-development/fdc00f514c7f24074ba8': failed to update object (Update for <http://flyte.lyft.com/v1alpha1|flyte.lyft.com/v1alpha1>, Kind=FlyteWorkflow) managed fields: failed to update ManagedFields: failed to convert old object: <http://flyte.lyft.com/v1alpha1|flyte.lyft.com/v1alpha1>, Kind=FlyteWorkflow is unstructured and is not suitable for converting to ""

Do you know where that error is coming from? Sounds like the controller wants to update the workflow but something breaks. Could it be related to the cluster version?

This is really odd.

@purple-match-42006 this is unexpected. For a little context, k8s uses

ManagedFields

to define which entities are allowed to modify status data in a CRD. Flyte uses the FlyteWorkflow CRD to schedule and track workflow execution. As FlytePropeller is the only entity that access this CRD we choose to clear the

ManagedFields

to reduce the CRD size and improve performance. It seems there is an issue here where k8s is rejecting this updated.

Is this a version thing?

i'm able to repro locally on 1.18.2, but I believe managed fields were introduced in 1.18 so I'm doing a little bit of digging.

Ah, okay got it, thanks for the context @hallowed-mouse-14616. Meanwhile, I found this bug issue which seems to be related https://github.com/kubernetes/kubernetes/issues/91006 The mentioned fix was merged after the version we are running as it seems.

I just started reading through the same issue 😄

Very very luckily, we have an update to 1.25+ planned for next sprint anyways. So, to solve our problem, I could wait until that is done. Anyways, good to know that is related to the Kubernetes version. Thanks for the quick help 🙇

Perfect timing! AFAIK there are no issues and I know we have a number of users on 1.25+. If you run into anything we're certainly here to help out though.

Sounds good, thanks for the support.