https://flyte.org
Join Slack
Hey, I am trying to run a flytesnacks workflow on Flyte deployed on our self-managed Kubernetes clus...
# flyte-deployment
p
Hey, I am trying to run a flytesnacks workflow on Flyte deployed on our self-managed Kubernetes cluster (v1.18.2). The workflow executed by 
pyflyte run --remote
is in the 
Running
state in the console, a 
workflow
is created but in propeller there is an error logged:
Copy code
E1117 09:51:41.531924       1 workers.go:102] error syncing 'flytesnacks-development/fdc00f514c7f24074ba8': failed to update object (Update for <http://flyte.lyft.com/v1alpha1|flyte.lyft.com/v1alpha1>, Kind=FlyteWorkflow) managed fields: failed to update ManagedFields: failed to convert old object: <http://flyte.lyft.com/v1alpha1|flyte.lyft.com/v1alpha1>, Kind=FlyteWorkflow is unstructured and is not suitable for converting to ""
Do you know where that error is coming from? Sounds like the controller wants to update the workflow but something breaks. Could it be related to the cluster version?
f
This is really odd.
Unexpected- cc @hallowed-mouse-14616
h
@purple-match-42006 this is unexpected. For a little context, k8s uses 
ManagedFields
to define which entities are allowed to modify status data in a CRD. Flyte uses the FlyteWorkflow CRD to schedule and track workflow execution. As FlytePropeller is the only entity that access this CRD we choose to clear the 
ManagedFields
to reduce the CRD size and improve performance. It seems there is an issue here where k8s is rejecting this updated.
I am looking into this now.
f
Is this a version thing?
h
i'm able to repro locally on 1.18.2, but I believe managed fields were introduced in 1.18 so I'm doing a little bit of digging.
Can confirm this works fine on newer k8s versions.
p
Ah, okay got it, thanks for the context @hallowed-mouse-14616. Meanwhile, I found this bug issue which seems to be related https://github.com/kubernetes/kubernetes/issues/91006 The mentioned fix was merged after the version we are running as it seems.
h
I just started reading through the same issue 😄
☺️ 1
Interesting - so to fix this we have a few options it seems: (1) update the k8s cluster - is this a possibility for you? (2) add a flag to flytepropeller to disable the clearing of ManagedFields - I'm a little hesitant to do this because we would be adding an option to support old k8s versions that will soon be out-of-use.
p
Very very luckily, we have an update to 1.25+ planned for next sprint anyways. So, to solve our problem, I could wait until that is done. Anyways, good to know that is related to the Kubernetes version. Thanks for the quick help 🙇
BTW, are there any Kubernetes version related things in Flyte that I should be aware of when updating?
h
Perfect timing! AFAIK there are no issues and I know we have a number of users on 1.25+. If you run into anything we're certainly here to help out though.
🙏 1
p
Sounds good, thanks for the support.
👍 1
154 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by