https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
v

Victor Gustavo da Silva Oliveira

03/02/2023, 5:42 PM
Hi everybody I'm working with FlyteFiles between two sepparate tasks: One download the file, the other works with it. In local environment, using sandbox, it work as expected. However, when running in a K8s cluster, it gives me this error:
Copy code
[Errno 2] No such file or directory: '/tmp/flytep4bezuvh/local_flytekit/d8e1f470ac5ed7ac3141ab1b6d83a854/6W2MGqmQ.jpg'
Can anybody help me with this?
f

Fabio Grätz

03/02/2023, 5:57 PM
Is this in the task that uploaded the file or the one that downloaded it again?
If in the latter, did you explicitly call 
.download()
?
v

Victor Gustavo da Silva Oliveira

03/02/2023, 5:59 PM
It's separated tasks In the first one I call 
.download()
. Second one I'm just trying to use the 
.path
to load the file. In this case, using PIL to load image
f

Fabio Grätz

03/02/2023, 6:00 PM
Can you please try the following?
Copy code
@task
def upload_file() -> FlyteFile:
    # Save image to <some_path>
    return FlyteFile(path=<some_path>)

@task
def download_file(f: FlyteFile) -> None:
    f.download()
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:01 PM
I forgot to mention, I'm getting the file from GCP. Let me share some snippets
I'm creating an list of FlyteFiles in the first task, using
Copy code
output.append(FlyteFile(path=f"gs://{bucket_name}/{img_prefix_path}"))
Then, in second task, I simply download it
Copy code
@task(cache=True, cache_version="1.0")
def load_file_as_flytefile(file: FlyteFile) -> FlyteFile:
    return file.download()
Then, finally, on the last task, I try to read it using PIL
Copy code
image = Image.open(samples[i].path).convert('RGB')
Sorry, but I cannot give more of the code. I hope this can help
f

Fabio Grätz

03/02/2023, 6:07 PM
Does the first task return a single FlyteFile or a list of them?
(Since the 2nd task appears to get only a single one)
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:07 PM
A list, then I download each using a map_task
f

Fabio Grätz

03/02/2023, 6:07 PM
Got it
If you download the image in the second task, it won’t be available in the 3rd because it runs in its own container
You could omit the 2nd task and call 
.download()
at the beginning of the 3rd task
Also, this caught my eye, not sure if it works: 
return file.download()
Maybe replace with
Copy code
file.download()
return file
(The 3rd task will have to download again regardless of this though)
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:13 PM
So, each task must download the file? I can't see what is the advantage of using FlyteFile, though...
f

Fabio Grätz

03/02/2023, 6:14 PM
So, each task must download the file?
Yes, this is correct 🙂 Each task runs in its own pod in the kubernetes cluster which could potentially be on a different node than the previous task.
The benefit of the 
FlyteFile
is that it takes care of the serialization and deserialization (to and from a bucket) when passing around files between tasks. Otherwise, you would have to take care of this (de-)serialization yourself.
Is the second task “doing a lot of work”? Otherwise I would recommend you omit the 2nd task and instead download the file directly in the 3rd task.
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:18 PM
It's not, but I intended to use the first ones as reference tasks in others workflows
f

Fabio Grätz

03/02/2023, 6:21 PM
Is the 2nd task doing something else than downloading the image?
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:22 PM
Nope. But we thoght that it will be better for our code if the latter tasks do not handle the download part, just deal with the manipulation
f

Fabio Grätz

03/02/2023, 6:26 PM
Ok understood. In Flyte, each task gets executed in a pod in kubernetes (think a container). These pods can run on completely different nodes (think virtual machines). This means that the tasks do not share a file system into which the 2nd task could download the image and the 3rd task will find it there. With other workflow orchestration engines, the user can only pass around strings between tasks. Flyte on the other hand allows you to pass more than strings between tasks, e.g. files. This is very convenient but behind the scenes all the inputs to a task have to be downloaded from a database/blob storage and all the return values have to be uploaded since the tasks run in different pods which can be on different virtual machines. So all the data that you want to manipulate in the 3rd task needs to be downloaded there. If the 2nd task does this …
Copy code
@task
def downooad_image(f: FlyteFile):
   f.download()
   return f
… then the image is first downloaded but then uploaded again behind the scenes when calling 
return f
.
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:35 PM
Wow, that's great! But as you said, each task run in a separate pod, right? How come sometimes I get SIGKILL 9 from my cluster? I'm testing with just 6 small files, it should work fine. I was convinced that was the files downloaded previously in the second task...
I was trying to rerun the workflow
Maybe it's a question for another thread, but I believe there is a connection
f

Fabio Grätz

03/02/2023, 6:44 PM
You might get out-of-memory-killed.
Can you please try requesting more memory in the task that fails (see docs).
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:45 PM
That's what I thought, but I was afraid it was something related as the previous error. I will try to change resources. Thanks, Fabio!
f

Fabio Grätz

03/02/2023, 6:49 PM
You’re welcome 🙂
Feel free to ping me in a thread in case you have more problems.
Have a nice evening/day!
v

Victor Gustavo da Silva Oliveira

03/02/2023, 6:49 PM
Thanks! Same to you!
3 Views
Powered by