hallowed-doctor-67759
02/22/2023, 6:48 PMhallowed-doctor-67759
02/22/2023, 6:48 PMflyte-backend-flyte-binary-webhook.flyte.svc
.
{
"json": {
"exec_id": "ap69gmldszgqd5xc94rk",
"node": "n0",
"ns": "...",
"res_ver": "208312048",
"routine": "worker-2",
"tasktype": "python-task",
"wf": "..."
},
"level": "error",
"msg": "Failed to launch job, system error. err: Internal error occurred: failed calling webhook \"<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>\": Post \"<https://flyte-backend-flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>\": service \"flyte-backend-flyte-binary-webhook\" not found",
"ts": "2023-02-22T18:26:12Z"
}
hallowed-doctor-67759
02/22/2023, 6:48 PMwebhook
section of the core.yaml
configmap for the propeller has
webhook:
certDir: /etc/webhook/certs
serviceName: flyte-pod-webhook
hallowed-doctor-67759
02/22/2023, 6:48 PMflyte-binary
deployment. But I eventually tore it down.hallowed-doctor-67759
02/22/2023, 6:50 PMhallowed-doctor-67759
02/22/2023, 6:50 PMflyte-backend-flyte-binary-webhook.flyte.svc
looks like the host names kubefwd
uses and I did use that tool at one point so I wonder if there was some weird alchemy that got things mixed up.average-finland-92144
02/22/2023, 7:10 PMhallowed-doctor-67759
02/22/2023, 7:13 PMaverage-finland-92144
02/22/2023, 7:17 PMflyte-binary
chart?
We should try a clean install if possiblehallowed-doctor-67759
02/22/2023, 7:18 PMhallowed-doctor-67759
02/22/2023, 7:26 PMhallowed-doctor-67759
02/22/2023, 7:26 PMhelm uninstall
. Then went through and cleared out the namespacesaverage-finland-92144
02/22/2023, 7:32 PMflyte-pod-webhook.flyte.svc
right?hallowed-doctor-67759
02/22/2023, 7:36 PMcore.yaml
in the running pod:
/etc/flyte/config $ cat core.yaml
manager:
pod-application: flytepropeller
pod-template-container-name: flytepropeller
pod-template-name: flytepropeller-template
propeller:
downstream-eval-duration: 30s
enable-admin-launcher: true
gc-interval: 12h
kube-client-config:
burst: 25
qps: 100
timeout: 30s
leader-election:
enabled: true
lease-duration: 15s
lock-config-map:
name: propeller-leader
namespace: flyte
renew-deadline: 10s
retry-period: 2s
limit-namespace: all
max-workflow-retries: 50
metadata-prefix: metadata/propeller
metrics-prefix: flyte
prof-port: 10254
queue:
batch-size: -1
batching-interval: 2s
queue:
base-delay: 5s
capacity: 1000
max-delay: 120s
rate: 100
type: maxof
sub-queue:
capacity: 1000
rate: 100
type: bucket
type: batch
rawoutput-prefix: <s3://infima-flyte/raw/>
workers: 40
workflow-reeval-duration: 30s
webhook:
certDir: /etc/webhook/certs
serviceName: flyte-pod-webhook
/etc/flyte/config $
hallowed-doctor-67759
02/22/2023, 7:36 PMhallowed-doctor-67759
02/22/2023, 8:18 PMhelm fetch --untar --untardir . flyteorg/flyte-core
2. Made two changes to the flyteadmin
and clusterresourcesync
deployment.yaml
. To get the cluster config working, had to add in these these volume mounts to the spec:
{{- with .Values.flyteadmin.additionalVolumeMounts -}}
{{ tpl (toYaml .) $ | nindent 10 }}
{{- end }}
3. Ran through the deploy, ie
helm upgrade flyte \
./flyte-core \
--install \
--values values.yaml \
--values values-eks.yaml \
--values values-cluster-config.yaml \
--values values-ingress.yaml \
--create-namespace \
--namespace flyte
Caveat: I start with the data plane and do some secret updating. But it’s basically that commandhallowed-doctor-67759
02/22/2023, 8:19 PMhallowed-doctor-67759
02/22/2023, 8:20 PM1.1.72
for flyteadmin
, are not the same as the tags that I get when I download the chart.
https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yamlhallowed-doctor-67759
02/22/2023, 8:20 PMflyteadmin:
enabled: true
# -- Replicas count for Flyteadmin deployment
replicaCount: 1
image:
# -- Docker image for Flyteadmin deployment
repository: <http://cr.flyte.org/flyteorg/flyteadmin-release|cr.flyte.org/flyteorg/flyteadmin-release> # FLYTEADMIN_IMAGE
tag: v1.3.0 # FLYTEADMIN_TAG
pullPolicy: IfNotPresent
hallowed-doctor-67759
02/22/2023, 8:21 PMhallowed-doctor-67759
02/22/2023, 8:42 PMvalues.yaml
to use the tags in the helm chart, so for instance
<http://cr.flyte.org/flyteorg/flytepropeller-release:v1.3.0|cr.flyte.org/flyteorg/flytepropeller-release:v1.3.0>
instead of
<http://cr.flyte.org/flyteorg/flytepropeller:v1.1.62|cr.flyte.org/flyteorg/flytepropeller:v1.1.62>
miniature-garden-94076
02/22/2023, 9:32 PMfreezing-airport-6809
freezing-boots-56761
freezing-boots-56761
freezing-boots-56761
freezing-boots-56761
hallowed-doctor-67759
02/23/2023, 2:49 AMok i think i understand. you had installed the flyte-binary chart before, but now uninstalling it and installing flyte-core?this is correct
hallowed-doctor-67759
02/23/2023, 2:49 AMi think you need to delete the old mutating webhook from the flyte namespacewhen I’ve been doing clean installs, i’ve completely wiped everything flyte related. the namespaces are all deleted.
freezing-boots-56761
hallowed-doctor-67759
02/23/2023, 2:53 AM╰─❯ kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-backend-flyte-binary-webhook 1 7d22h
flyte-pod-webhook 1 65d
hallowed-doctor-67759
02/23/2023, 2:53 AMfreezing-boots-56761
freezing-boots-56761
hallowed-doctor-67759
02/23/2023, 2:54 AMfreezing-boots-56761
freezing-airport-6809
freezing-boots-56761
hallowed-doctor-67759
02/23/2023, 2:57 AMhelm uninstall
doesn’t seem to tear down everythingfreezing-boots-56761
hallowed-doctor-67759
02/23/2023, 2:59 AMk get all -A
but clearly that wasn’t good enough.hallowed-doctor-67759
02/23/2023, 2:59 AMfreezing-boots-56761
freezing-boots-56761
hallowed-doctor-67759
02/23/2023, 3:07 AMkubectl api-resources --verbs=list -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found
hallowed-doctor-67759
02/23/2023, 3:11 AMfreezing-airport-6809
miniature-garden-94076
02/23/2023, 3:15 AMaverage-finland-92144
02/23/2023, 11:01 PMkubectl -n flyte port-forward *service/flyte-binary* 8088:8088 8089:8089
@hallowed-doctor-67759 would you like to contribute that piece? I can assist you if you needaverage-finland-92144
02/23/2023, 11:04 PMkubectl -n flyte port-forward *service/flyte-backend-flyte-binary* 8088:8088 8089:8089
freezing-boots-56761
hallowed-doctor-67759
02/24/2023, 5:42 AMaverage-finland-92144
02/24/2023, 1:20 PM