Hi again folks, I'm using

flyte-binary

deployed on k8s cluster hosted on DigitalOcean. It's been working great but recently I wanted to change the release name from "flyte-backend" to "flyte-binary". The rationale was simple - just wanted short service names (in hindsight, could have went with "nameOverride" in the values). After doing that, workflows fails to execute. The error was

Workflow[flyte-tasks:development:services.workflows.example.hello_world_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "flyte-pod-webhook.flyte.org": failed to call webhook: Post "<https://flyte-backend-flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": service "flyte-backend-flyte-binary-webhook" not found

Notice how it's trying to call old name of the service. Of course, I uninstalled the previous deployment, and reinstall with the new name. I can see the k8s service objects and pods with the correct new names. After spending several hours investigating it, re-deploying, uninstalling, deleting all of "flyte" namespaces (include the project+domain ones), deleting all the tables from the database - still the same damn error! It's unclear to me where it's getting old service name from. By now, I've pin-pointed that it only happens the task requests a secret. Any secret. And even more weird, after uninstalling the new deployment, deleting namespaces and dropping all tables in the database, installing Flyte with the previous release name "flyte-backend", now results in the same error, but the "new" service name:

Workflow[flytesnacks:development:hello_world.hello_world_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "flyte-pod-webhook.flyte.org": failed to call webhook: Post "<https://flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": service "flyte-binary-webhook" not found

I really don't get it. The configmaps or looks OK. I shelled into the pod and the config under

/etc/flyte/config.d

looks ok. After reverting back to the old name the

000-core.yaml

has the correct value under

webhook

section:

...
webhook:
  certDir: /var/run/flyte/certs
  localCert: true
  secretName: flyte-backend-flyte-binary-webhook-secret
  serviceName: flyte-backend-flyte-binary-webhook
  servicePort: 443
...

man! I was like checking the codebase and still not getting there. Glad you solved it!

I don't really know what do these webhooks are doing... and why is one configuration is selected over another. anyway to disable it altogether or is this important for something (events? metrics?)

I think it's used to read injected secrets. What if you just delete the old one?

webhooks is used to inject secrets yes.

@average-finland-92144 yes, deleted the old one and now everything works as expected (as i said at the start - resolved 🙂 ) What other ways do I have to consume secrets? Actually, all i need is a way to communicate information to a task that's not via parameters. And not during registration.

the bug is that helm uninstall doesn’t actually remove all components right?

Sure, I'll do that. Not really helm's fault because the webhook is created as part of service startup so not part deployment life cycle technically

maybe there’s a way to register it with helm though

It'd be difficult to articulate how grateful I was to find this thread today. Thanks @curved-petabyte-84246!

I was supposed to open an issue but didnt get around to it... 🤐