Alex Papanicolaou
02/22/2023, 6:48 PMflyte-backend-flyte-binary-webhook.flyte.svc
.
{
"json": {
"exec_id": "ap69gmldszgqd5xc94rk",
"node": "n0",
"ns": "...",
"res_ver": "208312048",
"routine": "worker-2",
"tasktype": "python-task",
"wf": "..."
},
"level": "error",
"msg": "Failed to launch job, system error. err: Internal error occurred: failed calling webhook \"<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>\": Post \"<https://flyte-backend-flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>\": service \"flyte-backend-flyte-binary-webhook\" not found",
"ts": "2023-02-22T18:26:12Z"
}
webhook
section of the core.yaml
configmap for the propeller has
webhook:
certDir: /etc/webhook/certs
serviceName: flyte-pod-webhook
flyte-binary
deployment. But I eventually tore it down.flyte-backend-flyte-binary-webhook.flyte.svc
looks like the host names kubefwd
uses and I did use that tool at one point so I wonder if there was some weird alchemy that got things mixed up.David Espejo (he/him)
02/22/2023, 7:10 PMAlex Papanicolaou
02/22/2023, 7:13 PMDavid Espejo (he/him)
02/22/2023, 7:17 PMflyte-binary
chart?
We should try a clean install if possibleAlex Papanicolaou
02/22/2023, 7:18 PMhelm uninstall
. Then went through and cleared out the namespacesDavid Espejo (he/him)
02/22/2023, 7:32 PMflyte-pod-webhook.flyte.svc
right?Alex Papanicolaou
02/22/2023, 7:36 PMcore.yaml
in the running pod:
/etc/flyte/config $ cat core.yaml
manager:
pod-application: flytepropeller
pod-template-container-name: flytepropeller
pod-template-name: flytepropeller-template
propeller:
downstream-eval-duration: 30s
enable-admin-launcher: true
gc-interval: 12h
kube-client-config:
burst: 25
qps: 100
timeout: 30s
leader-election:
enabled: true
lease-duration: 15s
lock-config-map:
name: propeller-leader
namespace: flyte
renew-deadline: 10s
retry-period: 2s
limit-namespace: all
max-workflow-retries: 50
metadata-prefix: metadata/propeller
metrics-prefix: flyte
prof-port: 10254
queue:
batch-size: -1
batching-interval: 2s
queue:
base-delay: 5s
capacity: 1000
max-delay: 120s
rate: 100
type: maxof
sub-queue:
capacity: 1000
rate: 100
type: bucket
type: batch
rawoutput-prefix: <s3://infima-flyte/raw/>
workers: 40
workflow-reeval-duration: 30s
webhook:
certDir: /etc/webhook/certs
serviceName: flyte-pod-webhook
/etc/flyte/config $
helm fetch --untar --untardir . flyteorg/flyte-core
2. Made two changes to the flyteadmin
and clusterresourcesync
deployment.yaml
. To get the cluster config working, had to add in these these volume mounts to the spec:
{{- with .Values.flyteadmin.additionalVolumeMounts -}}
{{ tpl (toYaml .) $ | nindent 10 }}
{{- end }}
3. Ran through the deploy, ie
helm upgrade flyte \
./flyte-core \
--install \
--values values.yaml \
--values values-eks.yaml \
--values values-cluster-config.yaml \
--values values-ingress.yaml \
--create-namespace \
--namespace flyte
Caveat: I start with the data plane and do some secret updating. But it’s basically that command1.1.72
for flyteadmin
, are not the same as the tags that I get when I download the chart.
https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yamlflyteadmin:
enabled: true
# -- Replicas count for Flyteadmin deployment
replicaCount: 1
image:
# -- Docker image for Flyteadmin deployment
repository: <http://cr.flyte.org/flyteorg/flyteadmin-release|cr.flyte.org/flyteorg/flyteadmin-release> # FLYTEADMIN_IMAGE
tag: v1.3.0 # FLYTEADMIN_TAG
pullPolicy: IfNotPresent
values.yaml
to use the tags in the helm chart, so for instance
<http://cr.flyte.org/flyteorg/flytepropeller-release:v1.3.0|cr.flyte.org/flyteorg/flytepropeller-release:v1.3.0>
instead of
<http://cr.flyte.org/flyteorg/flytepropeller:v1.1.62|cr.flyte.org/flyteorg/flytepropeller:v1.1.62>
Alex Papanicolaou
02/22/2023, 9:32 PMKetan (kumare3)
02/23/2023, 2:32 AMjeev
02/23/2023, 2:41 AMAlex Papanicolaou
02/23/2023, 2:49 AMok i think i understand. you had installed the flyte-binary chart before, but now uninstalling it and installing flyte-core?this is correct
i think you need to delete the old mutating webhook from the flyte namespacewhen I’ve been doing clean installs, i’ve completely wiped everything flyte related. the namespaces are all deleted.
jeev
02/23/2023, 2:52 AMAlex Papanicolaou
02/23/2023, 2:53 AM╰─❯ kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-backend-flyte-binary-webhook 1 7d22h
flyte-pod-webhook 1 65d
jeev
02/23/2023, 2:53 AMAlex Papanicolaou
02/23/2023, 2:54 AMjeev
02/23/2023, 2:55 AMKetan (kumare3)
02/23/2023, 2:55 AMjeev
02/23/2023, 2:56 AMAlex Papanicolaou
02/23/2023, 2:57 AMhelm uninstall
doesn’t seem to tear down everythingjeev
02/23/2023, 2:58 AMAlex Papanicolaou
02/23/2023, 2:59 AMk get all -A
but clearly that wasn’t good enough.jeev
02/23/2023, 3:00 AMAlex Papanicolaou
02/23/2023, 3:07 AMkubectl api-resources --verbs=list -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found
Ketan (kumare3)
02/23/2023, 3:14 AMAlex Papanicolaou
02/23/2023, 3:15 AMDavid Espejo (he/him)
02/23/2023, 11:01 PMkubectl -n flyte port-forward *service/flyte-binary* 8088:8088 8089:8089
@Alex Papanicolaou would you like to contribute that piece? I can assist you if you needkubectl -n flyte port-forward *service/flyte-backend-flyte-binary* 8088:8088 8089:8089
jeev
02/23/2023, 11:38 PMAlex Papanicolaou
02/24/2023, 5:42 AMDavid Espejo (he/him)
02/24/2023, 1:20 PM