https://flyte.org logo
Powered by
s

Stephan Gref

02/07/2022, 3:40 PM
Hi! I am playing around with raw container tasks and I kept getting the following error message:
Copy code
err failed when looking up Blob: Conf container:infrateam-dev-flyte-data != Passed Container:infrateam-dev-s3-source. Dynamic loading is disabled: not found"
although my flyte-data-config configmap looks like this:
Copy code
storage:
  type: s3
  connection:
    auth-type: iam
    region: my-region
  enable-multicontainer: true
In the end, I got it working by adding 
enable-multicontainer: true
to the propeller storage config. I looks like that the config map flyte-data-config is not used, although required by copilot.
y

Yuvraj

02/07/2022, 3:51 PM
Please read the config variable exist here https://github.com/flyteorg/flytestdlib/blob/master/storage/config.go#L55 If you are changing the default bucket name that is not define in config with flag 
container_name
(It's bucket name ). 
enable-multicontainer
will enable config in that you can use any bucket name.
s

Stephan Gref

02/07/2022, 3:53 PM
This is exactly what I configured, isn't it?
y

Yuvraj

02/07/2022, 3:59 PM
Can you provide me exact command of your copilot container ? We utilise propeller config for creating copilot command and passing it through cli https://github.com/flyteorg/flyteplugins/blob/ac44eae9a6a579a2bfce03c9fafbbbca8346c199/go/tasks/pluginmachinery/flytek8s/copilot.go#L61
s

Stephan Gref

02/07/2022, 4:01 PM
Copy code
initContainers:
    - name: flyte-copilot-downloader
      image: '<http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>'
      command:
        - /bin/flyte-copilot
        - '--storage.limits.maxDownloadMBs=0'
        - '--storage.type=s3'
        - '--storage.enable-multicontainer=false'
        - '--storage.container=<bucket-name>'
        - '--storage.connection.secret-key='
        - '--storage.connection.access-key='
        - '--storage.connection.auth-type=iam'
        - '--storage.connection.region=<region-name>'
        - '--storage.connection.endpoint='
Once I changed the config of propeller, 
--storage.enable-multicontainer=false
became 
true
y

Yuvraj

02/07/2022, 4:02 PM
Is it failing with 
--storage.enable-multicontainer=true
?
s

Stephan Gref

02/07/2022, 4:02 PM
No 🙂
But I suppose the command is constructed when propeller schedules the task - and it should respect the values from config map "flyte-data-config"
k

Ketan (kumare3)

02/07/2022, 4:22 PM
I think the command will override the config values
We have moved away from config to make it one less thing
7 Views
Powered by