first time flyte user, I have a ContainerTask which pulls image from AWS ECR and injects input like AWS S3 URI necessary for the task to run in command. My docker image has a subprocess command to call AWS S3 to download those URI file inputs but for some reason it returns CalledProcessError for the command and returned non-zero exit status 1 . Is there a better way to download input files from S3 into Flyte ContainerTask on runtime and also upload output back to S3. Can some please provide guidance on how can I debug and move ahead with this problem?

did you put AWS S3 URI in

flytefile

?

yes, FlyteFile is my data-type for s3 uri

I'm wondering that if there's some permission error

the error message could have been different I guess like access denied or something in calledprocess

does it work in local execution?

the process is too big and cannot run on local, so I have to push it to hosted flyte instance to see

maybe we have to give the

containertask1

more resource?

it has that already specified from pre-tested benchmarks.

how many resource did you give, especially memory?

cpu 4, mem 32Gi, nvidia.com/gpu: 1

so we have 3 container to execute the container task

nope, actually its just one single container task which is suppose to do all that in my case as I see start->container_task->end

@microscopic-animal-17045, are those s3 files in a bucket that your container has access to? If so, you can specify those files as inputs to the

ContainerTask

, something like:

consume_files = ContainerTask(
    name="my-name",
    input_data_dir="/var/inputs",
    inputs=kwtypes(file1=FlyteFile, file2=FlyteFile),
    image="<your-image>",
    command=[...],
)

and invoke

consume_files

specifying

file1

and

file2

. Flyte will make sure that those files will show up in the

/var/inputs/file1

and

/var/inputs/file2

respectively before your code runs.

hey @high-accountant-32689 thank you for the suggestion, above issue has been resolved.

@microscopic-animal-17045, can you share what was the final resolution?

sure, i switched to @task methods and used subprocess commands alongwith @vscode decorator to debug on failure cases. I leveraged FlyteFile download() method and direct aws s3 client commands for copying results back to S3. I had hardluck downloading FlyteFile in ContainerTask but when i switched to ShellTask it worked and later to just @task. Since ShellTask also has limitations on resource allocation attribute. My workflow required gpu and memory pre-defined so ultimately the basics helped me after all attempts.