curved-easter-24577
10/27/2023, 11:54 AMflyte-core
chart. We have defined our chart values as follow:
databaseSecret:
name: db-pass
volume:
- name: db-pass
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aws-db-secrets
secretManifest:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-db-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "<our secret arn>
jmesPath:
- path: password
objectAlias: dbpassword
But the _helpers.tpl
overrides the helm template with a volumes value that is not the required for AWS to work.
How we can avoid that? Do we need to change the _helpers.tpl
to add this option?
Thank you 🫶tall-lock-23197
curved-easter-24577
10/30/2023, 10:05 AMWarning FailedMount 26m (x371 over 3d) kubelet Unable to attach or mount volumes: unmounted volumes=[db-pass], unattached volumes=[kube-api-access-jgj6s aws-iam-token db-pass config-volume]: timed out waiting for the condition
curved-easter-24577
10/30/2023, 10:05 AMvolumes:
- name: db-pass
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aws-db-secrets
curved-easter-24577
10/30/2023, 10:06 AMvolumes:
- name: db-pass
secret:
secretName: db-pass
curved-easter-24577
10/30/2023, 10:07 AMtall-lock-23197
curved-easter-24577
10/30/2023, 3:41 PMdatabaseSecret:
name: db-pass
secretManifest:
apiVersion: <http://secrets-store.csi.x-k8s.io/v1|secrets-store.csi.x-k8s.io/v1>
kind: SecretProviderClass
metadata:
name: aws-db-secrets-spc
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.userSettings.db_secret }}"
objectType: "secretsmanager"
jmesPath:
- path: password
objectAlias: dbpassword
# Create k8s secret. It requires volume mount first in the pod and then sync.
secretObjects:
- secretName: db-pass
type: Opaque
data:
- objectName: dbpassword
key: dbpassword
With this the aws csi driver creats a base k8s secret and works 🤟average-finland-92144
10/30/2023, 3:59 PMdbpassword
secret ends up mounted as the db-pass
volume on flyteadmin
right?curved-easter-24577
10/30/2023, 4:12 PMcurved-easter-24577
10/30/2023, 8:00 PMdatacatalog:
# mount another db secret to activate secret-store-csi to create db-pass k8s secret.
additionalVolumes:
- name: aws-secret
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aws-db-secrets-spc
additionalVolumeMounts:
- name: aws-secret
mountPath: "/mnt/aws-secrets"
readOnly: true
curved-easter-24577
10/30/2023, 8:01 PMcurved-easter-24577
10/30/2023, 8:02 PMdatabaseSecret.volume
average-finland-92144
10/30/2023, 9:12 PMcurved-easter-24577
10/31/2023, 11:31 AMaverage-finland-92144
10/31/2023, 3:28 PMflyte-core
salmon-refrigerator-32115
01/06/2024, 1:25 AMaverage-finland-92144
01/09/2024, 3:31 PMsalmon-refrigerator-32115
01/09/2024, 10:13 PMcurved-easter-24577
01/10/2024, 8:27 AMsalmon-refrigerator-32115
01/10/2024, 5:10 PMcurved-easter-24577
01/11/2024, 8:15 AMsecret-store-csi-driver
and `secrets-provider-aws`:
# deploy secrets-store-csi-driver
- cd infrastructure/cluster/k8s_templates/secrets-store-csi-driver
- |
helm upgrade --install csi-secrets-store \
--namespace kube-system secrets-store-csi-driver/secrets-store-csi-driver \
-f values.yaml --version $SECRET_STORE_VERSION
- cd ../../../..
# deploy secrets-provider-aws
- |
helm upgrade --install secrets-provider-aws \
--namespace kube-system aws-secrets-manager/secrets-store-csi-driver-provider-aws \
--version $SECRET_PROVIDER_VERSION
Then configure the values of the secrets-store-csi-driver
as follwoing:
syncSecret:
enabled: true
enableSecretRotation: true
The secrets-store-csi-driver
will update the secret mounted in flyte automatically, you don’t need to restart flyte. Then configure Flyte-core chart values as the workaround on this Issue in the meantime.salmon-refrigerator-32115
01/12/2024, 11:26 PMsalmon-refrigerator-32115
01/12/2024, 11:27 PMaverage-finland-92144
01/15/2024, 7:52 PMsalmon-refrigerator-32115
01/16/2024, 6:46 PMWarning FailedMount 2s kubelet MountVolume.SetUp failed for volume "aws-secret" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod flyte/datacatalog-5df75c8455-ln79c, err: rpc error: code = Unknown │
│ desc = us-east-1: Failed fetching secret arn:aws:secretsmanager:us-east-1:178581358138:secret:database/service/mlforge/flyte/int/cyxd/aurorapostgres/v1/migrations-9YrdFD: WebIdentityErr: failed to retrieve credentials │
│ caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for <https://oidc.eks.us-east-1.amazonaws.com/id/235001DED36CBF377DFCFBF426B1CFE6> │
│ status code: 400, request id: cb368731-4d56-4649-822b-a89a03784cd9 │
average-finland-92144
01/16/2024, 6:53 PMaws eks describe-cluster --region <region> --name <Name-EKS-Cluster> --query "cluster.identity.oidc.issuer" --output text
curved-easter-24577
01/16/2024, 6:55 PMsalmon-refrigerator-32115
01/16/2024, 6:58 PM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::178581358138:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/235001DED36CBF377DFCFBF426B1CFE6"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<http://oidc.eks.us-east-1.amazonaws.com/id/235001DED36CBF377DFCFBF426B1CFE6:aud|oidc.eks.us-east-1.amazonaws.com/id/235001DED36CBF377DFCFBF426B1CFE6:aud>": "<http://sts.amazonaws.com|sts.amazonaws.com>"
}
}
}
]
}
salmon-refrigerator-32115
01/16/2024, 6:59 PMsalmon-refrigerator-32115
01/16/2024, 7:03 PM<https://oidc.eks.us-east-1.amazonaws.com/id/235001DED36CBF377DFCFBF426B1CFE6>
salmon-refrigerator-32115
01/16/2024, 7:04 PMcurved-easter-24577
01/16/2024, 7:16 PMsalmon-refrigerator-32115
01/16/2024, 9:51 PMWarning FailedMount 11s kubelet Unable to attach or mount volumes: unmounted volumes=[db-pass], unattached volumes=[db-pass config-volume kube-api-access-chn8t aws-iam-token aws-secret]: timed out waitin │
g for the condition
Warning FailedMount 7s (x9 over 2m14s) kubelet MountVolume.SetUp failed for volume "db-pass" : secret "db-pass" not found
salmon-refrigerator-32115
01/16/2024, 9:52 PMsalmon-refrigerator-32115
01/16/2024, 9:55 PMdatacatalog:
additionalVolumes:
- name: aws-secret
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: flyte-secretproviderclass
additionalVolumeMounts:
- name: aws-secret
mountPath: "/ect/aws-secrets"
readOnly: true
...
common:
### This secret needs to exist in the flyte namespace prior.
databaseSecret:
name: db-pass
secretManifest:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: flyte-secretproviderclass
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.userSettings.dbSecretArn }}" <- I used SM secret ARN
objectType: "secretsmanager"
jmesPath:
- path: password <- the secret json do have 'password' as a field.
objectAlias: dbpassword
# Create k8s secret. It requires volume mount first in the pod and then sync.
secretObjects:
- secretName: db-pass
type: Opaque
data:
- objectName: dbpassword
key: pass.txt <- this line is different from Marti's original.
salmon-refrigerator-32115
01/16/2024, 10:18 PMName: datacatalog-7cf54d8fc9-tjjhw
Namespace: flyte
Priority: 0
Service Account: datacatalog
Node: ip-100-72-230-230.ec2.internal/100.72.230.230
Start Time: Tue, 16 Jan 2024 14:15:48 -0800
Labels: <http://app.kubernetes.io/instance=flyte|app.kubernetes.io/instance=flyte>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=datacatalog|app.kubernetes.io/name=datacatalog>
<http://helm.sh/chart=flyte-core-3.0.5|helm.sh/chart=flyte-core-3.0.5>
pod-template-hash=7cf54d8fc9
Annotations: configChecksum: 50d02ef3537a5d9f111aa8b8db9e65061685a8d8d1c1c52fdef322c0491ed4a
<http://kubectl.kubernetes.io/restartedAt|kubectl.kubernetes.io/restartedAt>: 2024-01-16T14:15:48-08:00
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/datacatalog-7cf54d8fc9
Init Containers:
run-migrations:
Container ID:
Image: <http://cr.flyte.org/flyteorg/datacatalog:v1.9.37|cr.flyte.org/flyteorg/datacatalog:v1.9.37>
Image ID:
Port: <none>
Host Port: <none>
Command:
datacatalog
--config
/etc/datacatalog/config/*.yaml
migrate
run
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::178581358138:role/flyte-role
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/etc/datacatalog/config from config-volume (rw)
/etc/db from db-pass (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwf5l (ro)
Containers:
datacatalog:
Container ID:
Image: <http://cr.flyte.org/flyteorg/datacatalog:v1.9.37|cr.flyte.org/flyteorg/datacatalog:v1.9.37>
Image ID:
Ports: 8088/TCP, 8089/TCP, 10254/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
datacatalog
--config
/etc/datacatalog/config/*.yaml
serve
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
ephemeral-storage: 200Mi
memory: 500Mi
Requests:
cpu: 500m
ephemeral-storage: 200Mi
memory: 200Mi
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::178581358138:role/flyte-role
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/ect/aws-secrets from aws-secret (ro)
/etc/datacatalog/config from config-volume (rw)
/etc/db from db-pass (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwf5l (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
db-pass:
Type: Secret (a volume populated by a Secret)
SecretName: db-pass
Optional: false
shared-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: datacatalog-config
Optional: false
aws-secret:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: <http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>
FSType:
ReadOnly: true
VolumeAttributes: secretProviderClass=flyte-secretproviderclass
kube-api-access-gwf5l:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 42s default-scheduler Successfully assigned flyte/datacatalog-7cf54d8fc9-tjjhw to ip-100-72-230-230.ec2.internal
Warning FailedMount 10s (x7 over 41s) kubelet MountVolume.SetUp failed for volume "db-pass" : secret "db-pass" not found
salmon-refrigerator-32115
01/16/2024, 10:25 PM# Create k8s secret. It requires volume mount first in the pod and then sync.
secretObjects:
- secretName: db-pass
type: Opaque
data:
- objectName: dbpassword
key: dbpassword
Could you explain?salmon-refrigerator-32115
01/16/2024, 10:27 PMMountVolume.SetUp failed for volume "db-pass" : secret "db-pass" not found
is valid, since I changed the definition of common/databaseSecret/name: db-pass and the k8s secret named db-pass will not be created by the flyte charts.salmon-refrigerator-32115
01/16/2024, 10:38 PMsalmon-refrigerator-32115
01/17/2024, 12:11 AMapiVersion: apps/v1
kind: Deployment
metadata:
annotations:
<http://deployment.kubernetes.io/revision|deployment.kubernetes.io/revision>: "10"
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
creationTimestamp: "2023-08-11T17:44:54Z"
generation: 10
labels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
<http://app.kubernetes.io/name|app.kubernetes.io/name>: datacatalog
<http://helm.sh/chart|helm.sh/chart>: flyte-core-3.0.5
<http://helm.toolkit.fluxcd.io/name|helm.toolkit.fluxcd.io/name>: flyte
<http://helm.toolkit.fluxcd.io/namespace|helm.toolkit.fluxcd.io/namespace>: flyte
name: datacatalog
namespace: flyte
resourceVersion: "1309685335"
uid: 3c621765-3c6f-4078-a12e-735ec114fe71
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
<http://app.kubernetes.io/name|app.kubernetes.io/name>: datacatalog
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
configChecksum: 50d02ef3537a5d9f111aa8b8db9e65061685a8d8d1c1c52fdef322c0491ed4a
<http://kubectl.kubernetes.io/restartedAt|kubectl.kubernetes.io/restartedAt>: "2024-01-16T14:15:48-08:00"
creationTimestamp: null
labels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
<http://app.kubernetes.io/name|app.kubernetes.io/name>: datacatalog
<http://helm.sh/chart|helm.sh/chart>: flyte-core-3.0.5
spec:
containers:
- command:
- datacatalog
- --config
- /etc/datacatalog/config/*.yaml
- serve
image: <http://cr.flyte.org/flyteorg/datacatalog:v1.9.37|cr.flyte.org/flyteorg/datacatalog:v1.9.37>
imagePullPolicy: IfNotPresent
name: datacatalog
ports:
- containerPort: 8088
protocol: TCP
- containerPort: 8089
protocol: TCP
- containerPort: 10254
protocol: TCP
resources:
limits:
cpu: "1"
ephemeral-storage: 200Mi
memory: 500Mi
requests:
cpu: 500m
ephemeral-storage: 200Mi
memory: 200Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/db
name: db-pass
- mountPath: /etc/datacatalog/config
name: config-volume
- mountPath: /ect/aws-secrets
name: aws-secret
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- command:
- datacatalog
- --config
- /etc/datacatalog/config/*.yaml
- migrate
- run
image: <http://cr.flyte.org/flyteorg/datacatalog:v1.9.37|cr.flyte.org/flyteorg/datacatalog:v1.9.37>
imagePullPolicy: IfNotPresent
name: run-migrations
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/db
name: db-pass
- mountPath: /etc/datacatalog/config
name: config-volume
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
runAsUser: 1001
serviceAccount: datacatalog
serviceAccountName: datacatalog
terminationGracePeriodSeconds: 30
volumes:
- name: db-pass
secret:
defaultMode: 420
secretName: db-pass
- emptyDir: {}
name: shared-data
- configMap:
defaultMode: 420
name: datacatalog-config
name: config-volume
- csi:
driver: <http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>
readOnly: true
volumeAttributes:
secretProviderClass: flyte-secretproviderclass
name: aws-secret
status:
conditions:
- lastTransitionTime: "2023-08-11T17:44:54Z"
lastUpdateTime: "2023-08-11T17:44:54Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
- lastTransitionTime: "2024-01-16T22:25:49Z"
lastUpdateTime: "2024-01-16T22:25:49Z"
message: ReplicaSet "datacatalog-7cf54d8fc9" has timed out progressing.
reason: ProgressDeadlineExceeded
status: "False"
type: Progressing
observedGeneration: 10
replicas: 3
unavailableReplicas: 3
updatedReplicas: 1
salmon-refrigerator-32115
01/17/2024, 7:00 PMMountVolume.SetUp failed for volume "db-pass" : secret "db-pass" not found
seems obvious to me.
Because the charts originally was
common:
databaseSecret:
# -- Specify name of K8s Secret which contains Database password. Leave it empty if you don't need this Secret
# name: "db-pass"
name: "db-pass"
# -- Specify your Secret (with sensitive data) or pseudo-manifest (without sensitive data). See <https://github.com/godaddy/kubernetes-external-secrets>
secretManifest:
# -- Leave it empty if your secret already exists
# Else you can create your own secret object. You can use Kubernetes secrets, else you can configure external secrets
# For external secrets please install Necessary dependencies, like, of your choice
# - <https://github.com/hashicorp/vault>
# - <https://github.com/godaddy/kubernetes-external-secrets>
apiVersion: v1
kind: Secret
metadata:
name: db-pass
type: Opaque
stringData:
# -- If using plain text you can provide the password here
pass.txt: "{{ .Values.userSettings.dbPassword }}"
And the result from the secret.yaml template was to create a secret named db-pass: (from kind: Secret)
{{- with .Values.common.databaseSecret.secretManifest }}
{{ tpl (toYaml .) $ }}
{{- end }}
Now the code changed to :
common:
### This secret needs to exist in the flyte namespace prior.
databaseSecret:
name: db-pass
secretManifest:
apiVersion: <http://secrets-store.csi.x-k8s.io/v1|secrets-store.csi.x-k8s.io/v1>
kind: SecretProviderClass
metadata:
name: flyte-secretproviderclass
spec:
provider: aws
parameters:
objects: |
- objectName: "{{ .Values.userSettings.dbSecretArn }}"
objectType: "secretsmanager"
jmesPath:
- path: password
objectAlias: dbpassword
# Create k8s secret. It requires volume mount first in the pod and then sync.
secretObjects:
- secretName: db-pass
type: Opaque
data:
- objectName: dbpassword
key: dbpassword
It’s not creating a k8s secret. (from kind: SecretProviderClass).
Therefore in the pod, the secret db-pass was deleted by the deployment somehow.
@curved-easter-24577, could you check your system to see if secret was created and NOT deleted when you deploy your new code?