https://flyte.org logo
Join Slack
Powered by
Hi All ! I am trying to deploy flyte using the hel...
# flyte-support
q
Hi All ! I am trying to deploy flyte using the helm chart. My requirement is that I want to use s3 storage from the ceph/rook cluster that is running on the same Kubernetes cluster where flyte is being deployed. What are the required configuration parameters for this in the 
values.yaml
file ? is there a sample config snippet for using an s3 on premise instead of those from the cloud vendors ? googling shows many different options and some are confusing or even conflicting(like is it to be configured under 
datacatalog
or 
storage
section ? what is the right parameter for Access Key ID ? 
accessKy
or 
accessKeyID
? ). Any help or pointers to the right documentation would be greatly appreciated. Thanks ! P.S: I tried various options and it seems to be not working as trying to run sample workflows result in the following error,
Copy code
Error:%!(EXTRA *awserr.baseError=NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)
f
the config basically has to match this - https://github.com/flyteorg/flyte/blob/a2331bd69b9f9ac38ce4f292c11d2374d93960db/flytestdlib/storage/config.go#L33 so it can be s3, gcs, afs etc etc. Samples values file here
a
q
Thanks indeed @freezing-airport-6809 and @average-finland-92144! I went through these docs again and tried. A few things, 1. The above example `values.yaml`Ketan pointed is for 
flyte-binary
and I trying to deploy 
flyte
package, which is having a different structure for 
values.yaml
file, I understand. And when tried to follow the link/file it was still looking at default minio deployment, which was disabled in my case. 2. When given the following parameters in my 
values.yaml
deployed, its giving error,
Copy code
storage:
    ## metadataContainer: "flyte"
    ## userDataContainer: "flyte"
    # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
    type: s3
    # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
    ## bucketName: my-bucket1-943d7647-df7b-415e-b44b-67c6a2fac060
    # -- settings for storage type s3
    bucketName: "flyte"
    s3:
        region: "us-east-1"
        authType: "iam"  # tried 'accessKey' too
        connection:
          disableSSL: true  # Set to false if using HTTPS
          v2Signing: false
          accessKey: "XXXXXXXXXXX"  # Access key for Ceph/Rook S3
          secretKey: "XXXXXXXX"  # Secret key for Ceph/Rook S3
          endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    #container: "flyte"  # Name of the bucket in Ceph/Rook S3
    enableMultiContainer: false
3. The config map created is having this portion,
Copy code
storage.yaml: |
    storage:
      type: s3
      container: "flyte"
      connection:
        auth-type: iam
        region: us-east-1
      enable-multicontainer: false
It gives the following error when workflow is executed,
Copy code
grpc_message:"failed to create a signed url. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see 
aws.Config.CredentialsChainVerboseErrors"
So where/how should I give the 
accessKey
and 
secretKey
in 
values.yaml
to get it right ? Thanks again !
a
@quick-sunset-60703 what chart are you trying to install? is it flyte-core?
q
Copy code
apiVersion: v2
name: flyte
description: A Helm chart for Flyte Sandbox
type: application
version: v0.1.10  # VERSION
dependencies:
  - name: flyte-core
    alias: flyte
    version: v0.1.10  # VERSION
    repository: file://../flyte-core  # REPOSITORY
  - name: contour
    version: 7.10.1
    repository: <https://charts.bitnami.com/bitnami>
    condition: contour.enabled
  - name: spark-operator
    alias: sparkoperator
    version: 1.1.15
    repository: <https://kubeflow.github.io/spark-operator>
    condition: sparkoperator.enabled
  - name: dask-kubernetes-operator
    alias: daskoperator
    version: 2022.12.0
    repository: <https://helm.dask.org>
    condition: daskoperator.enabled
  - name: kubernetes-dashboard
    version: 4.0.2
    repository: <https://kubernetes.github.io/dashboard/>
    condition: kubernetes-dashboard.enabled
a
ok, the sandbox chart is intended to be used by `flytectl demo start`and it ships with all dependencies packed. If you want to customize for a specific blob storage platform etc you should go with 
flyte-binary
. The guide I linked takes you through the process for an on prem environment
q
Thanks ! had seen that link. But 
flyte-binary
has all the components of flyte bundled to single binary right ? Is there a way/chart that I can use to deploy flyte with all the components as separate(admin, console, properller etc.) ?
Can I use 
core
?
a
yeah, that's the one for that use case
q
Thanks indeed. Lemme try that !
Hi @average-finland-92144 I deployed 
flyte-core
with the following config for storage,
Copy code
storage:
  # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
  type: s3
  # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
  bucketName: "flyte"
  # -- settings for storage type s3
  s3:
    region: "us-east-1"
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    # -- type of authentication to use for S3 buckets, can either be iam or accesskey
    authType: "accesskey"
    # -- AWS IAM user access key ID to use for S3 bucket auth, only used if authType is set to accesskey
    accessKey: "XXXX access key to use for S3 bucket auth, only used if authType is set to accesskey
    secretKey: "XXXXXX"
And the 
datacatalog-config
configmap has the following entries,
Copy code
storage.yaml: |
    storage:
      type: minio
      container: "my-s3-bucket"
      stow:
        kind: s3
        config:
          access_key_id: minio
          auth_type: accesskey
          secret_key: miniostorage
          disable_ssl: true
          endpoint: <http://minio.flyte.svc.cluster.local:9000>
          region: us-east-1
How do I disable minio and make it use my ceph/rook s3 bucket named 
flyte
?
I checked the template files, it looks fine. But whatever changes I make to the 
storage
section in 
values.yaml
is not having any impact and it always falls back to the 
minio
storage. template section,
Copy code
storage:
{{- if eq .Values.storage.type "s3" }}
  type: s3
  container: {{ .Values.storage.bucketName | quote }}
  connection:
    auth-type: {{ .Values.storage.s3.authType }}
    region: {{ .Values.storage.s3.region }}
    endpoint: {{ .Values.storage.s3.endpoint | quote }} 
    {{- if eq .Values.storage.s3.authType "accesskey" }}
    access-key: {{ .Values.storage.s3.accessKey }}
    secret-key: {{ .Values.storage.s3.secretKey }}
    {{- end }}
Ignore the above two messages by me as I was still (wrongly) using the "flyte" instead of "flyte-core" ! my bad. Still, I haven't got it deployed successfully. But thats a different issue and I am working on it. Many thanks indeed for the help !
@average-finland-92144, here is the problem. I am able to deploy the chart and bring up all the pods successfully using 
flyte-deps
and 
flyte-core
. But not able to use ceph/rook because nowhere in the chart's 
values.yaml
, there is an option to give 
endpoint
for the s3 object storage gateway. I can't see any mention of 
endpoint
in the template files too, when 
s3
is the selected storage backend. Any clue ? Thanks again !
a
@quick-sunset-60703 for on-prem storage you can use the 
sandbox
storage type and provide the endpoint and credentials See reference: https://github.com/flyteorg/flyte/blob/ae455650ffb7614f026cf07981a7b4a816448ea6/charts/flyte-core/templates/_helpers.tpl#L213-L224
no prob if it's not minio
q
Thanks for the reply @average-finland-92144! I had tried the suggested approach earlier and failed. Even after hard coding the endpont and credentials in the template file and pointing to 
sandbox
in 
values.yaml
, its not taking the given value and instead falls in to the default minio parameters. Please find the details below, 1. Relevant portion of 
values.yaml
Copy code
storage:
  type: sandbox
  bucketName: flyte
  s3:
    region: us-east-1
    authType: accesskey
    accessKey: "XXXXXXXXXXXXXXXX"
    secretKey: "XXXXXXXXXXXXX"
  enableMultiContainer: false
2. Portion of 
_helpers.tpl
Copy code
{{- else if eq .Values.storage.type "sandbox" }}
  type: minio
  container: {{ .Values.storage.bucketName | quote }}
  stow:
    kind: s3
    config:
      access_key_id: XXXXXXXXXXXXX
      auth_type: accesskey
      secret_key: XXXXXXXXXXXXXXXXXX
      disable_ssl: true
      endpoint: <http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>
      region: us-east-1
  signedUrl:
    stowConfigOverride:
      endpoint: <http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>
3. The command 
helm template flyte flyteorg/flyte-core -n flyte -f values.yaml
generate the manifest and it has the following part for storage,
Copy code
storage.yaml: |
    storage:
      type: minio
      container: "flyte"
      stow:
        kind: s3
        config:
          access_key_id: minio
          auth_type: accesskey
          secret_key: miniostorage
          disable_ssl: true
          endpoint: <http://minio.flyte.svc.cluster.local:9000>
          region: us-east-1
      signedUrl:
        stowConfigOverride:
          endpoint: <http://minio.flyte.svc.cluster.local:9000>
As you can see, it's still taking the default minio config. Am I doing something very stupid here ?
OK, using the command 
helm install flyte-core . -n flyte -f values.yaml
to explicitly use the modified template from local source, takes the endpoint and credentials correctly, as verified in the 
storage.yaml
section of 
datacatalog-config
configmap. Now will try to run the workflows and update here later. Many thanks indeed for all the help so far ! @average-finland-92144
a
oh you're right, when using the 
sandbox
type the endpoint url assumes minio. You would need to use the 
custom
storage type to prevent the helper to alter the structure
any further question let me know
q
So, you mean to say that in the 
values.yaml
file, I need to give the following ?
Copy code
storage:
    type: custom
Now, if chose 
custom
, how/where do I give the 
endpoint
and credentials in 
values.yaml
? also, do I need to add anything to 
_helpers.tpl
or modify it? Thanks again ! appreciate your persistence 🙂
a
no need to add anything to helpers 
custom
will take whatever config you add there. So you could use the same format as for the 
sandbox
config, just making sure your endpoint is right and test the template rendering to verify
q
Thanks @average-finland-92144 ! Please see the below snippets, 1. From 
values.yaml
Copy code
storage:
  type: custom
  custom:
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    accessKey: "XXXXXXXXXXXXXXX"
    secretKey: "XXXXXXXXXX"
    region: "us-east-1"
    insecure: true
    bucket: "flyte"
2. From 
_helpers.tpl
Copy code
{{- else if eq .Values.storage.type "custom" }}
{{- with .Values.storage.custom -}}
  {{ tpl (toYaml .) $ | nindent 2 }}
{{- end }}
{{- end }}
{{- end }}
3. From the generated manifest of 
datacatalog-config
(or similar configMap with storage section)
Copy code
storage.yaml: |
    storage:
      enable-multicontainer: false
      limits:
        maxDownloadMBs: 10
      cache:
        max_size_mbs: 0
        target_gc_percent: 70
Why is the template file not picking up the parameters from values.yaml file and rendering the storage specific config in the ConfigMaps ?
a
@quick-sunset-60703 I couldn't reproduce this. I used this config in the values for flyte-core with a minikube cluster:
Copy code
storage:
  type: custom    
  custom: 
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    accessKey: "ABCDE"
    secretKey: "FGHI"
    region: "us-east-1"
    insecure: true
    bucket: "flyte"
and this is, for example, the rendered flyte-propeller configmap:
Copy code
storage.yaml: | 
    storage:
      accessKey: ABCDE
      bucket: flyte
      endpoint: <http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>
      insecure: true
      region: us-east-1
      secretKey: FGHI
      enable-multicontainer: false
      limits:
        maxDownloadMBs: 10
      cache:
        max_size_mbs: 0
        target_gc_percent: 70
q
Thanks again @average-finland-92144! I got this particular error fixed. It seems to have been due to the 
gcs
entry taking precedence over the 
custom
entry in the 
values.yaml
file. Commenting out the 
gcs
entry fixed the issue and now the configmap is correct with the required parameters present. But now, I am facing a different issue. The `datacatalog pod is crashing and giving the following error in logs,
Copy code
root@node1:~/flyte/flyte/charts/flyte-core# kubectl logs -n flyte datacatalog-6b466d6484-jgjqw 
Defaulted container "datacatalog" out of: datacatalog, run-migrations (init)
time="2025-04-07T16:11:52Z" level=info msg="Using config file: [/etc/datacatalog/config/db.yaml /etc/datacatalog/config/server.yaml /etc/datacatalog/config/storage.yaml]"
{"json":{"app_name":"datacatalog"},"level":"error","msg":"Failed to create DataStore &{s3 {{{      false false   }} iam   us-east-1 false} { map[]}  false {0 70} {10} {map[] 0s} {map[]}}, err initContainer is required even with `enable-multicontainer`","ts":"2025-04-07T16:11:52Z"}
{"json":{},"level":"fatal","msg":"caught panic: initContainer is required even with `enable-multicontainer` [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:88 +0x7a\npanic({0x1ef78e0?, 0xc000476900?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:96 +0x851\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc000b1fb50|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc000b1fb50>?, 0x48397c?}, 0xc000190300)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:144 +0x1d8\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318>, 0x3cc09c0}, 0xc000190300)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:122 +0x37\<http://ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc0007d2300|ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc0007d2300>?, {0x22ab220?, 0x4?, 0x22ab108?})\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/serve.go:54 +0x37a\<http://ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80|ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80>, {0xc00068efa0, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x882\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)|ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5\<http://ngithub.com/spf13/cobra.(*Command).Execute(...)|ngithub.com/spf13/cobra.(*Command).Execute(...)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46 +0x1a\nmain.main()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/main.go:11 +0x6c\n]","ts":"2025-04-07T16:11:52Z"}
root@node1:~/flyte/flyte/charts/flyte-core#
Any clue ? Thanks !
a
hm seem to be complaining about creating a datastore for datacatalog do you plane to use caching? otherwise that service is not needed. Can you share the contents of your 
storage
Helm values section?
q
Thanks again @average-finland-92144.Here is the 
storage
section,
Copy code
storage:
  type: custom
  custom:
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    accessKey: "XXXXXXXXXXXXXXXXXXXX"
    secretKey: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    region: "us-east-1"
    insecure: true
    bucket: "flyte"
  enableMultiContainer: false
  # -- default limits being applied to storage config
  limits:
    maxDownloadMBs: 10
  cache:
    maxSizeMBs: 0
    targetGCPercent: 70
No, I don't need caching for now. I just wanna get the basic flow working and will think of everything else later 🙂
a
let's try with disabling datacatalog for now just set the following in your values file
Copy code
datacatalog:
  enabled: false
q
Thanks again @average-finland-92144 ! Had tried that and obviously the error with datacatalog disappears, but similar error persists with 
propellor
and 
flyteadmin
,
Copy code
root@node1:~/flyte/flyte/charts/flyte-core# kubectl logs -n flyte flyteadmin-ff99db5b8-pzbbp
Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), sync-cluster-resources (init), generate-secrets (init)
time="2025-04-10T03:50:27Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{},"level":"error","msg":"Failed to initialize storage config","ts":"2025-04-10T03:50:33Z"}
panic: initContainer is required even with `enable-multicontainer`

goroutine 1 [running]:
github.com/flyteorg/flyte/flyteadmin/pkg/server.newGRPCServer({0x357d058, 0x4f0c4c0}, 0xc0009b4640, 0xc000a78180, 0xc000526000, {0x0, 0x0}, {0x35a6548, 0xc000fae5e0}, {0x3549c40, ...}, ...)
	/go/src/github.com/flyteorg/flyteadmin/pkg/server/service.go:149 +0x122f
github.com/flyteorg/flyte/flyteadmin/pkg/server.serveGatewayInsecure({0x357d058, 0x4f0c4c0}, 0xc0009b4640, 0xc000a78180, 0xc000a7ac08, 0xc000526000, 0x0, {0x35a6548, 0xc000fae5e0})
	/go/src/github.com/flyteorg/flyteadmin/pkg/server/service.go:377 +0x632
github.com/flyteorg/flyte/flyteadmin/pkg/server.Serve({0x357d058, 0x4f0c4c0}, 0xc0009b4640, 0x0)
	/go/src/github.com/flyteorg/flyteadmin/pkg/server/service.go:67 +0x195
github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7(0xc000ba8900?, {0x2d3300e?, 0x4?, 0x2d32eee?})
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/serve.go:45 +0x275
github.com/spf13/cobra.(*Command).execute(0x4e829e0, {0xc00097f7c0, 0x2, 0x2})
	/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x882
github.com/spf13/cobra.(*Command).ExecuteC(0x4e82700)
	/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.Execute(0x60?)
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/root.go:49 +0x35
main.main()
	/go/src/github.com/flyteorg/flyteadmin/cmd/main.go:12 +0x78
root@node1:~/flyte/flyte/charts/flyte-core# 
root@node1:~/flyte/flyte/charts/flyte-core# 
root@node1:~/flyte/flyte/charts/flyte-core# kubectl logs -n flyte flytepropeller-95cdd5ffc-2stkr 
time="2025-04-10T03:50:24Z" level=info msg=------------------------------------------------------------------------
time="2025-04-10T03:50:24Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2025-04-10 03:50:23.05611712 +0000 UTC m=+0.002686734]"
time="2025-04-10T03:50:24Z" level=info msg=------------------------------------------------------------------------
time="2025-04-10T03:50:24Z" level=info msg="Detected: 10 CPU's\n"
{"json":{},"level":"fatal","msg":"Failed to start controller. Error: failed to start FlytePropeller: Failed to create Metadata storage: initContainer is required even with `enable-multicontainer`","ts":"2025-04-10T03:50:34Z"}
root@node1:~/flyte/flyte/charts/flyte-core# 
root@node1:~/flyte/flyte/charts/flyte-core#
a
got it, that makes sense so looking at the `stow`library that Flyte uses to build the storage client, I'd try to go with the following config:
Copy code
storage:
  type: custom
  bucketName: "flyte"
  custom:
    auth_type: accesskey
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    access_key_id: "XXXXXXXXXXXXXXXXXXXX"
    secret_key_id: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    region: "us-east-1"
    disable_ssl = "true"
q
values.yaml,
Copy code
storage:
  type: custom
  bucketName: "flyte"
  custom:
    auth_type: accesskey
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    access_key_id: "XXXXXXXXXXXXXXXX"
    secret_key_id: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    region: "us-east-1"
    disable_ssl: true
  enableMultiContainer: false
and here is the 
datacatalog
crashing with the same error,
Copy code
root@node1:~# kubectl logs -n flyte datacatalog-65fd89688d-wwbhv
Defaulted container "datacatalog" out of: datacatalog, run-migrations (init)
time="2025-04-10T16:27:35Z" level=info msg="Using config file: [/etc/datacatalog/config/db.yaml /etc/datacatalog/config/server.yaml /etc/datacatalog/config/storage.yaml]"
{"json":{"app_name":"datacatalog"},"level":"error","msg":"Failed to create DataStore &{s3 {{{      false false   }} iam   us-east-1 false} { map[]}  false {0 70} {10} {map[] 0s} {map[]}}, err initContainer is required even with `enable-multicontainer`","ts":"2025-04-10T16:27:35Z"}
{"json":{},"level":"fatal","msg":"caught panic: initContainer is required even with `enable-multicontainer` [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:88 +0x7a\npanic({0x1ef78e0?, 0xc000a5e490?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:96 +0x851\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc000b1fb50|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc000b1fb50>?, 0x48397c?}, 0xc00029e2a0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:144 +0x1d8\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318>, 0x3cc09c0}, 0xc00029e2a0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:122 +0x37\<http://ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc000777500|ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc000777500>?, {0x22ab220?, 0x4?, 0x22ab108?})\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/serve.go:54 +0x37a\<http://ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80|ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80>, {0xc0007e13c0, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x882\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)|ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5\<http://ngithub.com/spf13/cobra.(*Command).Execute(...)|ngithub.com/spf13/cobra.(*Command).Execute(...)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46 +0x1a\nmain.main()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/main.go:11 +0x6c\n]","ts":"2025-04-10T16:27:35Z"}
root@node1:~#
a
can you drop that 
enableMultiContainer
line?
q
sure. Thanks again !
Copy code
storage:
  type: custom
  bucketName: "flyte"
  custom:
    auth_type: accesskey
    endpoint: "<http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>"  # Ceph/Rook S3 endpoint
    access_key_id: "XXXXXXXXXXXXXX"
    secret_key_id: "XXXXXXXXXXXXXXXXXXXX"
    region: "us-east-1"
    disable_ssl: true
#enableMultiContainer: false
Still the same error though,
Copy code
root@node1:~# kubectl logs -n flyte datacatalog-5cb994847c-9jqd9
Defaulted container "datacatalog" out of: datacatalog, run-migrations (init)
time="2025-04-10T16:30:46Z" level=info msg="Using config file: [/etc/datacatalog/config/db.yaml /etc/datacatalog/config/server.yaml /etc/datacatalog/config/storage.yaml]"
{"json":{"app_name":"datacatalog"},"level":"error","msg":"Failed to create DataStore &{s3 {{{      false false   }} iam   us-east-1 false} { map[]}  false {0 70} {10} {map[] 0s} {map[]}}, err initContainer is required even with `enable-multicontainer`","ts":"2025-04-10T16:30:47Z"}
{"json":{},"level":"fatal","msg":"caught panic: initContainer is required even with `enable-multicontainer` [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:88 +0x7a\npanic({0x1ef78e0?, 0xc0009a5630?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()>\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:96 +0x851\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00089fb50|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00089fb50>?, 0x48397c?}, 0xc000992c30)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:144 +0x1d8\<http://ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318|ngithub.com/flyteorg/flyte/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x29cd318>, 0x3cc09c0}, 0xc000992c30)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:122 +0x37\<http://ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc000297b00|ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.init.func3(0xc000297b00>?, {0x22ab220?, 0x4?, 0x22ab108?})\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/serve.go:54 +0x37a\<http://ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80|ngithub.com/spf13/cobra.(*Command).execute(0x3c40e80>, {0xc00057c360, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x882\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)|ngithub.com/spf13/cobra.(*Command).ExecuteC(0x3c40ba0)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5\<http://ngithub.com/spf13/cobra.(*Command).Execute(...)|ngithub.com/spf13/cobra.(*Command).Execute(...)>\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/flyteorg/flyte/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46 +0x1a\nmain.main()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/main.go:11 +0x6c\n]","ts":"2025-04-10T16:30:47Z"}
root@node1:~#
a
can you describe what lands in the configmap? Like 
kubectl describe cm flytepropeller-config -n flyte
not 100% sure about the cm name
q
seems to be fine for me,
Copy code
root@node1:~# kubectl get cm -n flyte flyte-propeller-config -o yaml
apiVersion: v1
data:
  admin.yaml: |
    admin:
      clientId: 'flytepropeller'
      clientSecretLocation: /etc/secrets/client_secret
      endpoint: flyteadmin:81
      insecure: true
    event:
      capacity: 1000
      rate: 500
      type: admin
  catalog.yaml: |
    catalog-cache:
      endpoint: datacatalog:89
      insecure: true
      type: datacatalog
  copilot.yaml: |
    plugins:
      k8s:
        co-pilot:
          image: cr.flyte.org/flyteorg/flytecopilot-release:v1.15.0
          name: flyte-copilot-
          start-timeout: 30s
  core.yaml: |
    manager:
      pod-application: flytepropeller
      pod-template-container-name: flytepropeller
      pod-template-name: flytepropeller-template
    propeller:
      downstream-eval-duration: 30s
      enable-admin-launcher: true
      leader-election:
        enabled: true
        lease-duration: 15s
        lock-config-map:
          name: propeller-leader
          namespace: flyte
        renew-deadline: 10s
        retry-period: 2s
      limit-namespace: all
      literal-offloading-config:
        enabled: false
      max-workflow-retries: 30
      metadata-prefix: metadata/propeller
      metrics-prefix: flyte
      prof-port: 10254
      queue:
        batch-size: -1
        batching-interval: 2s
        queue:
          base-delay: 5s
          capacity: 1000
          max-delay: 120s
          rate: 100
          type: maxof
        sub-queue:
          capacity: 100
          rate: 10
          type: bucket
        type: batch
      rawoutput-prefix: <s3://my-s3-bucket/>
      workers: 4
      workflow-reeval-duration: 30s
    webhook:
      certDir: /etc/webhook/certs
      serviceName: flyte-pod-webhook
  enabled_plugins.yaml: |
    tasks:
      task-plugins:
        default-for-task-types:
          container: container
          container_array: k8s-array
          sidecar: sidecar
        enabled-plugins:
        - container
        - sidecar
        - k8s-array
        - agent-service
        - echo
  k8s.yaml: |
    plugins:
      k8s:
        default-cpus: 100m
        default-env-vars: []
        default-memory: 100Mi
  resource_manager.yaml: |
    propeller:
      resourcemanager:
        type: noop
  storage.yaml: "storage:\n  access_key_id: XXXXXXXXXXXXXXXXX\n  auth_type: accesskey\n
    \ disable_ssl: true\n  endpoint: <http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local:80>\n
    \ region: us-east-1\n  secret_key_id: XXXXXXXX\n
    \ enable-multicontainer: \n  limits:\n    maxDownloadMBs: 10\n  cache:\n    max_size_mbs:
    0\n    target_gc_percent: 70\n"
  task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-enabled: false
        kubernetes-enabled: false
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: flyte-core
    meta.helm.sh/release-namespace: flyte
  creationTimestamp: "2025-04-10T16:30:12Z"
  labels:
    app.kubernetes.io/instance: flyte-core
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: flyteadmin
    helm.sh/chart: flyte-core-v1.15.0
  name: flyte-propeller-config
  namespace: flyte
  resourceVersion: "6733386"
  uid: 31fc2c75-3b9f-4a49-a695-1b77428b08fc
root@node1:~#
a
hm there's something weird in formatting
q
anything else to try @average-finland-92144 ?
Hi @average-finland-92144! I tried the latest version of the chart with tag 
1.15.3
of the images. The problem persists with exactly same errors. Do you want me to open a GitHub issue for this ?
33 Views
Open in Slack
PreviousNext
Powered by