I’ve deployed Flyte with the

flyte-binary

Helm chart on a GCP GKE cluster. I’ve been able to run workflows and tasks in the cluster. There were successful and failed executions. I can see that there are many pods created with status Error for those failed executions, and the pods stay there. If I delete those failed pods, the information about the executions seem to be vanished, and Flyte Console will complain that it “Failed to fetch data”. Is this an expected behavior that there are many pods created, failed and remained in my cluster? I believe it’s not, but I don’t really get how those failed pods are managed. Any enlightenment would really be appreciated

There is a configured garbage collection system. The default interval is 23 hours. Reason they are kept behind is for logs and debugging

@freezing-airport-6809 That explains it, thank you very much ☺️

You can adjust the interval too

I've wondered this for a while, thanks for explaining!

Hi @witty-planet-44210! I was curious which guide did you follow to deploy

flyte-binary

on GCP GKE ? I was able to deploy

flyte-binary

on EKS using the flyte-the-hard-way guide but it only has instructions for EKS - been looking for GCP GKE instructions but haven't found anything so far

Have you looked at the deployment guide

we dont have docs for this yet afaict.

Ohh that’s a miss

@fancy-napkin-28087 There is no documentations for deploying Flyte to GCP (omitted since v1.0.0). What I did was to look at the source code of the Helm chart and build my own values.yaml file. In my deployment, I didn’t use GCS but MinIO deployed on the same cluster. It took a while to figure out everything. I’m not sure if I have deployed Flyte properly on GCP yet 😄 But so far, it’s been working for me.

@freezing-airport-6809 @freezing-boots-56761 is there any chance that you can open source something like a

gcp-starter.yaml

values file ?

@fancy-napkin-28087 yes

Yes we will

should just be the storage and logging configs mostly that are different. and ingress if you are using GCE LB

Yup

I can help unblock you quickly if you have specific questions @fancy-napkin-28087

We run on GCP but not with single binary. A bit of a custom setup with GKE ingress controller and istio service mesh

We’re also not using the single binary and doing ingress via an istio service mesh.

@agreeable-kitchen-44189 May I ask if there is any reason for you to deploy the ingress via istio. I’ve not really worked with it but I want to know how it would be beneficial.

For is this was what we also use for our backend (non-flyte) service stack, so mostly for consistency reasons