boundless-sunset-37446
08/12/2022, 9:09 PMkubectl-n flytesnacks-development create secret generic common-secrets --from-literal=TEST_SECRET=blah
kubectl -n flytesnacks-development get secret/common-secrets -o json | jq '.data | to_entries | map(.value= (.value | @base64d))'
>> [
{
"key": "TEST_SECRET",
"value": "blah"
}
]
Within the flytesnacks-development namespace, I'm running a task like this
@task(secret_requests=[Secret(group="common-secrets", key="TEST_SECRET")])
def print_secret() -> str:
secrets = current_context().secrets
return secrets.get("common-secrets", "TEST_SECRET")
However, this fails with the following
Unable to find secret for key TEST_SECRET in group common-secrets in Env Var:_FSEC_COMMON-SECRETS_TEST_SECRET and FilePath: /root/secrets/common-secrets/test_secret
From looking at the code in the the SecretManager, it looks like it only checks the ENV variable or a file path (which does not exist because I'm using k8s secrets as an additional provider). Am I missing something? I've checked that the k8s secret is in the same namespace as the task being runfreezing-airport-6809
boundless-sunset-37446
08/15/2022, 1:42 AMfreezing-airport-6809
boundless-sunset-37446
08/15/2022, 2:41 PM➜ kubectl logs pod/flyte-pod-webhook-595f7b6858-7qxdt -n flyte
time="2022-08-12T00:31:32Z" level=info msg=------------------------------------------------------------------------
time="2022-08-12T00:31:32Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-08-12 00:31:32.456181799 +0000 UTC m=+0.033941479]"
time="2022-08-12T00:31:32Z" level=info msg=------------------------------------------------------------------------
time="2022-08-12T00:31:32Z" level=info msg="Detected: 4 CPU's\n"
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
freezing-airport-6809
boundless-sunset-37446
08/15/2022, 4:57 PM➜ Documents kb describe pod a2wf2sqs2bfw6qr4l92d-n0-0 -n flytesnacks-development
Name: a2wf2sqs2bfw6qr4l92d-n0-0
Namespace: flytesnacks-development
Priority: 0
Node: ip-10-15-147-196.ec2.internal/10.15.147.196
Start Time: Mon, 15 Aug 2022 12:52:24 -0400
Labels: domain=development
execution-id=a2wf2sqs2bfw6qr4l92d
inject-flyte-secrets=true
interruptible=false
node-id=n0
project=flytesnacks
shard-key=13
task-name=flyte-workflows-hello-world-print-secret
workflow-name=flyte-workflows-hello-world-wf
Annotations: <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
flyte.secrets/s0: m4zg54lqhiqcey2pnvww52rnonswg3tforzsectlmv3tuibclfavercjl4jukq1sivkcecq
<http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status: Succeeded
IP: 10.15.128.66
IPs:
IP: 10.15.128.66
Controlled By: flyteworkflow/a2wf2sqs2bfw6qr4l92d
Containers:
a2wf2sqs2bfw6qr4l92d-n0-0:
Container ID: <docker://7e388ada4103f5fa7c5a0c1a673b3c61f79c441bcede9b1798bed8f4db128e6>5
Image: XXX
Image ID: XXX
Port: <none>
Host Port: <none>
Args:
pyflyte-execute
--inputs
<s3://predictap-tyson-flyte/metadata/propeller/flytesnacks-development-a2wf2sqs2bfw6qr4l92d/n0/data/inputs.pb>
--output-prefix
<s3://predictap-tyson-flyte/metadata/propeller/flytesnacks-development-a2wf2sqs2bfw6qr4l92d/n0/data/0>
--raw-output-data-prefix
<s3://predictap-tyson-flyte/jh/a2wf2sqs2bfw6qr4l92d-n0-0>
--checkpoint-path
<s3://predictap-tyson-flyte/jh/a2wf2sqs2bfw6qr4l92d-n0-0/_flytecheckpoints>
--prev-checkpoint
""
--resolver
flytekit.core.python_auto_container.default_task_resolver
--
task-module
flyte.workflows.hello_world
task-name
print_secret
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 15 Aug 2022 12:52:25 -0400
Finished: Mon, 15 Aug 2022 12:52:28 -0400
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 1000Mi
Requests:
cpu: 1
memory: 1000Mi
Environment:
FLYTE_INTERNAL_EXECUTION_WORKFLOW: flytesnacks:development:<http://flyte.workflows.hello_world.wf|flyte.workflows.hello_world.wf>
FLYTE_INTERNAL_EXECUTION_ID: a2wf2sqs2bfw6qr4l92d
FLYTE_INTERNAL_EXECUTION_PROJECT: flytesnacks
FLYTE_INTERNAL_EXECUTION_DOMAIN: development
FLYTE_ATTEMPT_NUMBER: 0
FLYTE_INTERNAL_TASK_PROJECT: flytesnacks
FLYTE_INTERNAL_TASK_DOMAIN: development
FLYTE_INTERNAL_TASK_NAME: flyte.workflows.hello_world.print_secret
FLYTE_INTERNAL_TASK_VERSION: v0.0.5
FLYTE_INTERNAL_PROJECT: flytesnacks
FLYTE_INTERNAL_DOMAIN: development
FLYTE_INTERNAL_NAME: flyte.workflows.hello_world.print_secret
FLYTE_INTERNAL_VERSION: v0.0.5
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: XXX
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7kr64 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
kube-api-access-7kr64:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
freezing-airport-6809
flyte.secrets/s0: m4zg54lqhiqcey2pnvww52rnonswg3tforzsectlmv3tuibclfavercjl4jukq1sivkcecq
freezing-airport-6809
-
a problem?thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
freezing-airport-6809
thankful-minister-83577
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 6m29s (x28 over 106m) kubelet Unable to attach or mount volumes: unmounted volumes=[mnxw12lpnywxgzldojsxi316], unattached volumes=[kube-api-access-pbzq2 mnxw12lpnywxgzldojsxi316 aws-iam-token]: timed out waiting for the condition
Warning FailedMount 2m13s (x60 over 108m) kubelet MountVolume.SetUp failed for volume "mnxw12lpnywxgzldojsxi316" : references non-existent secret key: test_secret
thankful-minister-83577
boundless-sunset-37446
08/15/2022, 11:29 PMboundless-sunset-37446
08/16/2022, 2:34 AM➜ symphony_hall git:(main) ✗ kubectl -n flytesnacks-development get secret/common-secrets -o json | jq '.data | to_entries | map(.value= (.value | @base64d))'
[
{
"key": "TEST_SECRET",
"value": "blah"
},
{
"key": "test_secret",
"value": "some_value"
}
]
Code change
@task(secret_requests=[Secret(group="common-secrets", key="test_secret")])
def print_secret() -> str:
secrets = current_context().secrets
return secrets.get("common-secrets", "test_secret")
but still got the same error
Unable to find secret for key test_secret in group common-secrets in Env Var:_FSEC_COMMON-SECRETS_TEST_SECRET and FilePath: /root/secrets/common-secrets/test_secret
I wonder if the -
in common-secrets
is causing issues. I also changed the secrets default directory in the Dockerfile. Could that cause problems? I know the default is /etc/secrets
ENV FLYTE_SECRETS_DEFAULT_DIR /root/secrets
thankful-minister-83577
$ alias ksd
ksd='kubectl -n flytesnacks-development'
$ ksd create secret generic common-secrets --from-literal=test-secret=sosecret
secret/common-secrets created
with code (assuming the flytesnacks repo in cookbook/)
$ cat core/secret_example.py
from flytekit import Secret, task, workflow, current_context
@task(secret_requests=[Secret(group="common-secrets", key="test-secret")])
def print_secret():
secrets = current_context().secrets
s = secrets.get("common-secrets", "test-secret")
print(s)
@workflow
def my_print():
print_secret()
$ ksd logs f0cc52c90e3ae47a8876-n0-0
tar: Removing leading `/' from member names
sosecret
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
boundless-sunset-37446
08/16/2022, 12:52 PMboundless-sunset-37446
08/16/2022, 1:16 PMboundless-sunset-37446
08/16/2022, 1:18 PMroot@alxtm7pdc9zchkqttvtp-n0-0:~# printenv
KUBERNETES_SERVICE_PORT_HTTPS=443
FLYTE_INTERNAL_EXECUTION_DOMAIN=development
PYTHON_VERSION=3.9.13
FLYTE_INTERNAL_TASK_PROJECT=flytesnacks
FLYTE_INTERNAL_VERSION=v0.0.12
FLYTE_INTERNAL_PROJECT=flytesnacks
FLYTE_INTERNAL_TASK_NAME=flyte.workflows.hello_world.print_secret
FLYTE_SECRETS_DEFAULT_DIR=/root/secrets
FLYTE_INTERNAL_TASK_DOMAIN=development
FLYTE_INTERNAL_EXECUTION_PROJECT=flytesnacks
thankful-minister-83577
ls -lRa /root/secrets
thankful-minister-83577
boundless-sunset-37446
08/16/2022, 4:26 PMboundless-sunset-37446
08/16/2022, 4:27 PM/root/secrets
didn't get created since the secrets weren't moved overthankful-minister-83577
thankful-minister-83577
boundless-sunset-37446
08/16/2022, 4:39 PMhallowed-mouse-14616
08/17/2022, 6:45 PMboundless-sunset-37446
08/17/2022, 7:01 PMboundless-sunset-37446
08/17/2022, 7:15 PMclever-vegetable-6086
08/17/2022, 7:17 PMclever-vegetable-6086
08/17/2022, 7:18 PMfreezing-airport-6809
TEST_SECRET
clever-vegetable-6086
08/17/2022, 8:44 PMtest_secret
. case issue should be quick to verify on our end.freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
boundless-sunset-37446
08/17/2022, 8:47 PMhallowed-mouse-14616
08/17/2022, 8:47 PMhallowed-mouse-14616
08/17/2022, 8:48 PMfailurePolicy: Fail
boundless-sunset-37446
08/17/2022, 8:50 PMhallowed-mouse-14616
08/17/2022, 8:50 PMWorkflow[flytesnacks:development:.flytegen.core.containerization.use_secrets.secret_task] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[corecontainerizationusesecretssecrettask]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.all.svc:9443/mutate--v1-pod?timeout=10s>": service "flyte-pod-webhook" not found
but i just removed the webhook service - this should tell us if the webhook is active.thankful-minister-83577
hallowed-mouse-14616
08/17/2022, 8:52 PMthankful-minister-83577
clever-vegetable-6086
08/17/2022, 9:11 PMfailurePolicy
? i thought the mutating webhook config was at the service level, but not seeing it there?hallowed-mouse-14616
08/17/2022, 9:12 PMkubectl -n flyte edit mutatingwebhookconfigurations flyte-pod-webhook -o yaml
clever-vegetable-6086
08/17/2022, 9:12 PMclever-vegetable-6086
08/17/2022, 9:12 PMboundless-sunset-37446
08/17/2022, 9:13 PMhallowed-mouse-14616
08/17/2022, 9:15 PMclever-vegetable-6086
08/17/2022, 9:15 PMFail
just nowclever-vegetable-6086
08/17/2022, 9:16 PMIgnore
previously 🙂hallowed-mouse-14616
08/17/2022, 9:16 PMboundless-sunset-37446
08/17/2022, 9:16 PMhallowed-mouse-14616
08/17/2022, 9:18 PMhallowed-mouse-14616
08/17/2022, 9:19 PMhallowed-mouse-14616
08/17/2022, 9:21 PMfailed calling webhook
? on my local testing (above) the propeller logs are littered with:
{
"json": {
"exec_id": "agbb24545n6vbwknrxqs",
"node": "corecontainerizationusesecretssecrettask",
"ns": "flytesnacks-development",
"res_ver": "5504",
"routine": "worker-4",
"src": "handler.go:222",
"wf": "flytesnacks:development:.flytegen.core.containerization.use_secrets.secret_task"
},
"level": "error",
"msg": "handling parent node failed with error: failed at Node[corecontainerizationusesecretssecrettask]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook \"<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>\": Post \"<https://flyte-pod-webhook.all.svc:9443/mutate--v1-pod?timeout=10s>\": service \"flyte-pod-webhook\" not found",
"ts": "2022-08-17T15:45:24-05:00"
}
for each individual retry.clever-vegetable-6086
08/17/2022, 9:23 PMclever-vegetable-6086
08/17/2022, 9:23 PMclever-vegetable-6086
08/17/2022, 9:24 PM{"json":{"exec_id":"ancq4kjld5lf6cv88bd8","ns":"flytesnacks-development","res_ver":"10680755","routine":"worker-8","wf":"flytesnacks:development:<http://flyte.workflows.hello_world.wf|flyte.workflows.hello_world.wf>"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook \"<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>\": failed to call webhook: Post \"<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>\": context deadline exceeded]. Error Type[*errors.NodeErrorWithCause]","ts":"2022-08-17T21:22:16Z"}
E0817 21:22:16.167997 1 workers.go:102] error syncing 'flytesnacks-development/ancq4kjld5lf6cv88bd8': failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": context deadline exceed
hallowed-mouse-14616
08/17/2022, 9:32 PMboundless-sunset-37446
08/17/2022, 9:34 PMhallowed-mouse-14616
08/17/2022, 9:34 PMboundless-sunset-37446
08/17/2022, 9:34 PMhallowed-mouse-14616
08/17/2022, 9:36 PMapiVersion: <http://admissionregistration.k8s.io/v1|admissionregistration.k8s.io/v1>
kind: MutatingWebhookConfiguration
metadata:
// ommitted
webhooks:
// omitted
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
hallowed-mouse-14616
08/17/2022, 9:37 PMservice:
name: flyte-pod-webhook
namespace: all
path: /mutate--v1-pod
port: 9443
clever-vegetable-6086
08/17/2022, 9:38 PM<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod>
?clever-vegetable-6086
08/17/2022, 9:38 PMclever-vegetable-6086
08/17/2022, 9:42 PMkubectl exec -n flyte -it flytepropeller-74bf956f6c-5zfbh -- /bin/sh
nc -z flyte-pod-webhook.flyte.svc 443
hallowed-mouse-14616
08/17/2022, 9:53 PMclever-vegetable-6086
08/17/2022, 9:54 PMhallowed-mouse-14616
08/17/2022, 9:54 PMapiVersion: v1
kind: ConfigMap
metadata:
name: flyte-propeller-config
namespace: flyte
and the pod-webhook service:
apiVersion: v1
kind: Service
metadata:
name: flyte-pod-webhook
namespace: flyte
clever-vegetable-6086
08/17/2022, 9:54 PMclever-vegetable-6086
08/17/2022, 9:56 PMkind: ConfigMap
metadata:
annotations:
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
creationTimestamp: "2022-08-12T00:31:18Z"
labels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
<http://app.kubernetes.io/name|app.kubernetes.io/name>: flyteadmin
<http://helm.sh/chart|helm.sh/chart>: flyte-core-v1.1.0
name: flyte-propeller-config
namespace: flyte
clever-vegetable-6086
08/17/2022, 9:57 PMapiVersion: v1
kind: Service
metadata:
annotations:
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
<http://projectcontour.io/upstream-protocol.h2c|projectcontour.io/upstream-protocol.h2c>: grpc
creationTimestamp: "2022-08-12T00:31:18Z"
labels:
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
name: flyte-pod-webhook
namespace: flyte
hallowed-mouse-14616
08/17/2022, 9:57 PMclever-vegetable-6086
08/17/2022, 9:57 PMhallowed-mouse-14616
08/17/2022, 9:57 PMhallowed-mouse-14616
08/17/2022, 9:57 PMclever-vegetable-6086
08/17/2022, 9:59 PMkubectl get svc/flyte-pod-webhook -o yaml -n flyt
apiVersion: v1
kind: Service
metadata:
annotations:
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
<http://projectcontour.io/upstream-protocol.h2c|projectcontour.io/upstream-protocol.h2c>: grpc
creationTimestamp: "2022-08-12T00:31:18Z"
labels:
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
name: flyte-pod-webhook
namespace: flyte
resourceVersion: "8525789"
uid: 0ffd7a94-14d1-44db-ae76-34d17dccb0cd
spec:
clusterIP: 172.27.91.248
clusterIPs:
- 172.27.91.248
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: https
port: 443
protocol: TCP
targetPort: 9443
selector:
app: flyte-pod-webhook
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
clever-vegetable-6086
08/17/2022, 9:59 PMhallowed-mouse-14616
08/17/2022, 9:59 PMhallowed-mouse-14616
08/17/2022, 10:02 PMservice:
name: flyte-pod-webhook
namespace: all
path: /mutate--v1-pod
port: 9443
mutatingwebhookconfigurations flyte-pod-webhook
clever-vegetable-6086
08/17/2022, 10:02 PMclever-vegetable-6086
08/17/2022, 10:03 PMservice:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 443
to
service:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 9443
clever-vegetable-6086
08/17/2022, 10:03 PMhallowed-mouse-14616
08/17/2022, 10:06 PMclever-vegetable-6086
08/17/2022, 10:08 PMclever-vegetable-6086
08/17/2022, 10:08 PMboundless-sunset-37446
08/17/2022, 10:09 PM{"json":{"exec_id":"aknhc8ds5wn5z7sql6jb","ns":"flytesnacks-development","res_ver":"10692263","routine":"worker-5","wf":"flytesnacks:development:<http://flyte.workflows.hello_world.wf|flyte.workflows.hello_world.wf>"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook \"<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>\": failed to call webhook: Post \"<https://flyte-pod-webhook.flyte.svc:9443/mutate--v1-pod?timeout=10s>\": no service port 9443 found for service \"flyte-pod-webhook\"]. Error Type[*errors.NodeErrorWithCause]","ts":"2022-08-17T22:07:16Z"}
E0817 22:07:16.516247 1 workers.go:102] error syncing 'flytesnacks-development/aknhc8ds5wn5z7sql6jb': failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:9443/mutate--v1-pod?timeout=10s>": no service port 9443 found for service "flyte-pod-webhook"
{"json":{"exec_id":"ancq4kjld5lf6cv88bd8","ns":"flytesnacks-development","routine":"worker-4"},"level":"warning","msg":"Workflow not found in cache.","ts":"2022-08-17T22:08:06Z"}
{"json":{"exec_id":"ancq4kjld5lf6cv88bd8","ns":"flytesnacks-development","routine":"worker-4"},"level":"warning","msg":"Workflow namespace[flytesnacks-development]/name[ancq4kjld5lf6cv88bd8] not found, may be deleted.","ts":"2022-08-17T22:08:06Z"}
boundless-sunset-37446
08/17/2022, 10:09 PMhallowed-mouse-14616
08/17/2022, 10:11 PMhallowed-mouse-14616
08/17/2022, 10:11 PMclever-vegetable-6086
08/18/2022, 4:05 PMthankful-minister-83577
thankful-minister-83577
clever-vegetable-6086
08/18/2022, 4:23 PMthankful-minister-83577
clever-vegetable-6086
08/18/2022, 4:28 PMInternal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": context deadline exceed
• we were able to determine that it was an issue from the k8s API -> webhook pod by looking at this diagramclever-vegetable-6086
08/18/2022, 4:29 PMclever-vegetable-6086
08/18/2022, 4:30 PMhallowed-mouse-14616
08/18/2022, 4:42 PMhallowed-mouse-14616
08/18/2022, 4:43 PMclever-vegetable-6086
08/18/2022, 4:45 PM