Hey I am trying to run a flytesnacks workflow on Flyte deplo

Question

Hey I am trying to run a flytesnacks workflow on Flyte deployed on our self managed Kubernetes cluster v1 18 2 The workflow executed by `pyflyte run remote` is in the `Running` state in the console a `workflow` is created but in propeller there is an error logged ```E1117 09 51 41 531924 1 workers go 102 error syncing flytesnacks development fdc00f514c7f24074ba8 failed to update object Update for Kind=FlyteWorkflow managed fields failed to update ManagedFields failed to convert old object Kind=FlyteWorkflow is unstructured and is not suitable for converting to ``` Do you know where that error is coming from Sounds like the controller wants to update the workflow but something breaks Could it be related to the cluster version

Ketan (kumare3) · Answer

This is really odd

Ketan (kumare3) · Answer

Unexpected cc < Dan Rammer hamersaw >

Dan Rammer (hamersaw) · Answer

< Hanno Küpers> this is unexpected For a little context k8s uses `ManagedFields` to define which entities are allowed to modify status data in a CRD Flyte uses the FlyteWorkflow CRD to schedule and track workflow execution As FlytePropeller is the only entity that access this CRD we choose to clear the `ManagedFields` to reduce the CRD size and improve performance It seems there is an issue here where k8s is rejecting this updated

Dan Rammer (hamersaw) · Answer

I am looking into this now

Ketan (kumare3) · Answer

Is this a version thing

Dan Rammer (hamersaw) · Answer

i m able to repro locally on 1 18 2 but I believe managed fields were introduced in 1 18 so I m doing a little bit of digging

Dan Rammer (hamersaw) · Answer

Can confirm this works fine on newer k8s versions

Hanno Küpers · Answer

Ah okay got it thanks for the context < Dan Rammer hamersaw > Meanwhile I found this bug issue which seems to be related The mentioned fix was merged after the version we are running as it seems

Dan Rammer (hamersaw) · Answer

I just started reading through the same issue smile

Dan Rammer (hamersaw) · Answer

Interesting so to fix this we have a few options it seems 1 update the k8s cluster is this a possibility for you 2 add a flag to flytepropeller to disable the clearing of ManagedFields I m a little hesitant to do this because we would be adding an option to support old k8s versions that will soon be out of use

Hanno Küpers · Answer

Very very luckily we have an update to 1 25+ planned for next sprint anyways So to solve our problem I could wait until that is done Anyways good to know that is related to the Kubernetes version Thanks for the quick help bow

Hanno Küpers · Answer

BTW are there any Kubernetes version related things in Flyte that I should be aware of when updating

Dan Rammer (hamersaw) · Answer

Perfect timing AFAIK there are no issues and I know we have a number of users on 1 25+ If you run into anything we re certainly here to help out though

Hanno Küpers · Answer

Sounds good thanks for the support