https://flyte.org
Join Slack
secret was created
# flyte-support
n
secret was created
t
could you paste the full error message please? does it come with a file/line number?
n
the message is in gen-admin-auth-secret (init) logs on flyte-backend-flyte-binary pod
Copy code
time="2024-01-11T21:12:29Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
ERROR 2024/01/11 21:12:29 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY] map[snowflake:snowflake]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
time="2024-01-11T21:12:29Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
ERROR 2024/01/11 21:12:29 Could not cast sv to map[string]interface{}; key=%!s(MISSING), st=%!v(MISSING), tt=%!v(MISSING), sv=%!v(MISSING), tv=%!v(MISSING) default-for-task-types=[]interface {} map[string]interface {}=[map[container:container] map[container_array:K8S-ARRAY] map[snowflake:snowflake]] map[container:container container_array:k8s-array sidecar:sidecar]=<nil>
t
can you remove the 
-
in the 
default-for-task-types
map?
it’s 
map[string]string
in the code. i think some of the documentation is incorrect.
n
error went away but flye wont start
Normal Started 42s kubelet Started container flyte Warning Unhealthy 5s kubelet Liveness probe failed: Get "http://172.31.45.245:8088/healthcheck": dial tcp 172.31.45.2458088 connect: connection refused Warning Unhealthy 5s kubelet Readiness probe failed: Get "http://172.31.45.245:8088/healthcheck": dial tcp 172.31.45.2458088 connect: connection refused
t
can you get logs on the binary container?
n
yeah
Copy code
2024/01/11 22:23:09 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/finisher_api.go:509
[0.457ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2","annotations":null}}
last lines
t
it’s still borked?
can you do 
kubectl -n flyte get pod
n
flyte-backend-flyte-binary-656697c75b-bd7rw 0/1 Running 7 (3m19s ago) 12m
t
i think that should be fine
there were some restarts
n
not its in crashloop
Reason: Error Exit Code: 137
root@e486f538c2ef:~# kubectl -n flyte get pod NAME READY STATUS RESTARTS AGE flyte-backend-flyte-binary-656697c75b-bd7rw 0/1 CrashLoopBackOff 7 (2m21s ago) 16m
t
ah
can you do 
kubectl -n flyte logs flyte-backend-flyte-binary-656697c75b-bd7rw -p
n
ok - i found an issue
i had added an default-env-vars with a dash in it without escaping sreing
strange it gives no eror at all on that
this kills it
if you add quotes arounf the key - it works
t
k
so all better?
n
let me tyry to re-add snowflake plugin cfg
t
we could probably do a better job returning an error.
k
n
without "-"
dies again
these 3 lines (2 snowflake and 1 clientSecretsExternalSecretRef) is all i added
should i also "" the clientSecretsExternalSecretRef?
t
any more insight in the logs?
n
as soon as i add this line - it wont start anymore
no errors in the logs
t
check 
-p
logs
there should be some
n
whats' -p logs?
tahts what i am looking at
t
-p is previous, pulls logs from the crashed container
kubectl -n flyte logs podname -p
n
Copy code
[0.467ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"gcpSecretManager":{"sidecarImage":"<http://gcr.io/google.com/cloudsdktool/cloud-sdk:alpine|gcr.io/google.com/cloudsdktool/cloud-sdk:alpine>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2","annotations":null}}
same as current log
t
that looks like the migration logs.
or the secrets init container
can you 
describe
the pod?
we need to find the logs for the container that’s crashing.
kubectl -n flyte describe podname
n
it has 2 init containers and flyte in it....
t
can you 
-c containername
and look at all the containers individually?
if it’s crashing, there should be a pretty obvious log error
n
-c containername - ythats to what command?
t
logs
n
same log for flyte - its just a bunch of select statements
kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c flyte
these are 2 other logs
Copy code
root@e486f538c2ef:/app/copilot/flytepipe/admin# kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c wait-for-db
<http://flyteadmin.cluster-c9mjsckwrx0i.us-east-2.rds.amazonaws.com:5432|flyteadmin.cluster-c9mjsckwrx0i.us-east-2.rds.amazonaws.com:5432> - accepting connections
root@e486f538c2ef:/app/copilot/flytepipe/admin# kubectl -n flyte logs flyte-backend-flyte-binary-945c7f47b-mchdq -c gen-admin-auth-secret
time="2024-01-11T23:11:13Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
time="2024-01-11T23:11:14Z" level=info msg="Using config file: [/etc/flyte/config.d/000-core.yaml /etc/flyte/config.d/001-plugins.yaml /etc/flyte/config.d/002-database.yaml /etc/flyte/config.d/003-storage.yaml /etc/flyte/config.d/004-auth.yaml /etc/flyte/config.d/012-database-secrets.yaml /etc/flyte/config.d/014-auth-secrets.yaml /etc/flyte/config.d/100-inline-config.yaml]"
t
add -p also?
n
init containers have no -p
logs
flyte one looks the same - bunch of SELECT statments
t
are you able to ascertain which container is crashing?
n
flyte
it is strange that this bit you asked to remove "-" in
worked like this yesterday
but today only works without "-"
t
yeah i’m seeing it both ways in various places but the actual object in code is definitely just a map of str->str
can you add snowflake and comment out the container array stuff?
n
like this?
t
yeah
n
it works with array too
it's this line
that kilsl it
t
but we can’t find any error message?
and that secret exists?
kubectl -n flyte get secret flyte-binary-client-secrets-external-secret returns something?
n
root@e486f538c2ef:/app/copilot/flytepipe# kubectl -n flyte get secret flyte-binary-client-secrets-external-secret NAME TYPE DATA AGE flyte-binary-client-secrets-external-secret Opaque 1 7s
t
what is this secret for btw?
like what’s in it?
n
its a snowflake JWT-token
t
i see
where did you get that string from? clientSecretsExternalSecretRef i don’t think that’s the right one
was it in the snowflake guide? i’m not seeing it
n
t
you installed via helm right?
n
y
t
could you try something for me please?
let’s rename that secret.
Copy code
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: snowflake-secret
  namespace: flyte
type: Opaque
stringData:
  FLYTE_SNOWFLAKE_CLIENT_TOKEN: <JWT_TOKEN>
EOF
remove the secrets bit from the config, the part that was making it crash
and in your helm values file, include these values
Copy code
configuration:
  inlineSecretRef: snowflake-secret
or if you’re planning on adding more, maybe rename 
snowflake-secret
to `external-services`or something.
and helm apply
the 
clientSecretsExternalSecretRef
i think is the wrong value to set, that one has to do with auth. don’t want to touch auth
n
ok - will try
t
did it work?
n
actually got sidetracked on snowflake side - will try again soon
let me actually try with duymmy key
do i need to do
Copy code
configuration:
  inlineSecretRef: snowflake-secret
or if i already have - do i add it there?
Copy code
configuration:
   inline:
      secrets:
          adminOauthClientCredentials:
                enabled: true
                 clientSecret: Zt8RV
i tried like this
Copy code
configuration:
  inlineSecretRef: snowflake-secret
and it came up
i cant test snowflake right now but it is up
t
let me know when you get it working?
but it sounds like the apikey/secret bit is resolved at least right? i can update documentation today to reflect.
n
yes - looks like key issue is solved
10 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by