https://flyte.org
Join Slack
Hi! I am playing around with raw container tasks and I kept getting the following error message: ``...
# flyte-deployment
s
Hi! I am playing around with raw container tasks and I kept getting the following error message:
Copy code
err failed when looking up Blob: Conf container:infrateam-dev-flyte-data != Passed Container:infrateam-dev-s3-source. Dynamic loading is disabled: not found"
although my flyte-data-config configmap looks like this:
Copy code
storage:
  type: s3
  connection:
    auth-type: iam
    region: my-region
  enable-multicontainer: true
In the end, I got it working by adding 
enable-multicontainer: true
to the propeller storage config. I looks like that the config map flyte-data-config is not used, although required by copilot.
g
Please read the config variable exist here https://github.com/flyteorg/flytestdlib/blob/master/storage/config.go#L55 If you are changing the default bucket name that is not define in config with flag 
container_name
(It's bucket name ). 
enable-multicontainer
will enable config in that you can use any bucket name.
s
This is exactly what I configured, isn't it?
g
Can you provide me exact command of your copilot container ? We utilise propeller config for creating copilot command and passing it through cli https://github.com/flyteorg/flyteplugins/blob/ac44eae9a6a579a2bfce03c9fafbbbca8346c199/go/tasks/pluginmachinery/flytek8s/copilot.go#L61
s
Copy code
initContainers:
    - name: flyte-copilot-downloader
      image: '<http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>'
      command:
        - /bin/flyte-copilot
        - '--storage.limits.maxDownloadMBs=0'
        - '--storage.type=s3'
        - '--storage.enable-multicontainer=false'
        - '--storage.container=<bucket-name>'
        - '--storage.connection.secret-key='
        - '--storage.connection.access-key='
        - '--storage.connection.auth-type=iam'
        - '--storage.connection.region=<region-name>'
        - '--storage.connection.endpoint='
Once I changed the config of propeller, 
--storage.enable-multicontainer=false
became 
true
g
Is it failing with 
--storage.enable-multicontainer=true
?
s
No 🙂
But I suppose the command is constructed when propeller schedules the task - and it should respect the values from config map "flyte-data-config"
f
I think the command will override the config values
We have moved away from config to make it one less thing
235 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by