https://flyte.org
Join Slack
Hi everybody I'm working with FlyteFiles between two sepparate tasks: One download the file, the oth...
# flyte-support
d
Hi everybody I'm working with FlyteFiles between two sepparate tasks: One download the file, the other works with it. In local environment, using sandbox, it work as expected. However, when running in a K8s cluster, it gives me this error:
Copy code
[Errno 2] No such file or directory: '/tmp/flytep4bezuvh/local_flytekit/d8e1f470ac5ed7ac3141ab1b6d83a854/6W2MGqmQ.jpg'
Can anybody help me with this?
c
Is this in the task that uploaded the file or the one that downloaded it again?
If in the latter, did you explicitly call 
.download()
?
d
It's separated tasks In the first one I call 
.download()
. Second one I'm just trying to use the 
.path
to load the file. In this case, using PIL to load image
c
Can you please try the following?
Copy code
@task
def upload_file() -> FlyteFile:
    # Save image to <some_path>
    return FlyteFile(path=<some_path>)

@task
def download_file(f: FlyteFile) -> None:
    f.download()
d
I forgot to mention, I'm getting the file from GCP. Let me share some snippets
I'm creating an list of FlyteFiles in the first task, using
Copy code
output.append(FlyteFile(path=f"gs://{bucket_name}/{img_prefix_path}"))
Then, in second task, I simply download it
Copy code
@task(cache=True, cache_version="1.0")
def load_file_as_flytefile(file: FlyteFile) -> FlyteFile:
    return file.download()
Then, finally, on the last task, I try to read it using PIL
Copy code
image = Image.open(samples[i].path).convert('RGB')
Sorry, but I cannot give more of the code. I hope this can help
c
Does the first task return a single FlyteFile or a list of them?
(Since the 2nd task appears to get only a single one)
d
A list, then I download each using a map_task
c
Got it
If you download the image in the second task, it won’t be available in the 3rd because it runs in its own container
You could omit the 2nd task and call 
.download()
at the beginning of the 3rd task
Also, this caught my eye, not sure if it works: 
return file.download()
Maybe replace with
Copy code
file.download()
return file
(The 3rd task will have to download again regardless of this though)
d
So, each task must download the file? I can't see what is the advantage of using FlyteFile, though...
c
So, each task must download the file?
Yes, this is correct 🙂 Each task runs in its own pod in the kubernetes cluster which could potentially be on a different node than the previous task.
The benefit of the 
FlyteFile
is that it takes care of the serialization and deserialization (to and from a bucket) when passing around files between tasks. Otherwise, you would have to take care of this (de-)serialization yourself.
Is the second task “doing a lot of work”? Otherwise I would recommend you omit the 2nd task and instead download the file directly in the 3rd task.
d
It's not, but I intended to use the first ones as reference tasks in others workflows
c
Is the 2nd task doing something else than downloading the image?
d
Nope. But we thoght that it will be better for our code if the latter tasks do not handle the download part, just deal with the manipulation
c
Ok understood. In Flyte, each task gets executed in a pod in kubernetes (think a container). These pods can run on completely different nodes (think virtual machines). This means that the tasks do not share a file system into which the 2nd task could download the image and the 3rd task will find it there. With other workflow orchestration engines, the user can only pass around strings between tasks. Flyte on the other hand allows you to pass more than strings between tasks, e.g. files. This is very convenient but behind the scenes all the inputs to a task have to be downloaded from a database/blob storage and all the return values have to be uploaded since the tasks run in different pods which can be on different virtual machines. So all the data that you want to manipulate in the 3rd task needs to be downloaded there. If the 2nd task does this …
Copy code
@task
def downooad_image(f: FlyteFile):
   f.download()
   return f
… then the image is first downloaded but then uploaded again behind the scenes when calling 
return f
.
d
Wow, that's great! But as you said, each task run in a separate pod, right? How come sometimes I get SIGKILL 9 from my cluster? I'm testing with just 6 small files, it should work fine. I was convinced that was the files downloaded previously in the second task...
I was trying to rerun the workflow
Maybe it's a question for another thread, but I believe there is a connection
c
You might get out-of-memory-killed.
Can you please try requesting more memory in the task that fails (see docs).
d
That's what I thought, but I was afraid it was something related as the previous error. I will try to change resources. Thanks, Fabio!
c
You’re welcome 🙂
🙌 1
Feel free to ping me in a thread in case you have more problems.
Have a nice evening/day!
d
Thanks! Same to you!
191 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by