Hi everybody I m working with FlyteFiles between two seppara Flyte #ask-the-community

Hi everybody I'm working with FlyteFiles between t...

Victor Gustavo da Silva Oliveira

03/02/2023, 5:42 PM

Hi everybody I'm working with FlyteFiles between two sepparate tasks: One download the file, the other works with it. In local environment, using sandbox, it work as expected. However, when running in a K8s cluster, it gives me this error:

Copy code

[Errno 2] No such file or directory: '/tmp/flytep4bezuvh/local_flytekit/d8e1f470ac5ed7ac3141ab1b6d83a854/6W2MGqmQ.jpg'

Can anybody help me with this?

Fabio Grätz

03/02/2023, 5:57 PM

Is this in the task that uploaded the file or the one that downloaded it again?

Fabio Grätz

03/02/2023, 5:58 PM

If in the latter, did you explicitly call

.download()

Victor Gustavo da Silva Oliveira

03/02/2023, 5:59 PM

It's separated tasks In the first one I call

.download()

. Second one I'm just trying to use the

.path

to load the file. In this case, using PIL to load image

Fabio Grätz

03/02/2023, 6:00 PM

Can you please try the following?

Copy code

@task
def upload_file() -> FlyteFile:
    # Save image to <some_path>
    return FlyteFile(path=<some_path>)

@task
def download_file(f: FlyteFile) -> None:
    f.download()

Victor Gustavo da Silva Oliveira

03/02/2023, 6:01 PM

I forgot to mention, I'm getting the file from GCP. Let me share some snippets

Victor Gustavo da Silva Oliveira

03/02/2023, 6:05 PM

I'm creating an list of FlyteFiles in the first task, using

Copy code

output.append(FlyteFile(path=f"gs://{bucket_name}/{img_prefix_path}"))

Then, in second task, I simply download it

Copy code

@task(cache=True, cache_version="1.0")
def load_file_as_flytefile(file: FlyteFile) -> FlyteFile:
    return file.download()

Then, finally, on the last task, I try to read it using PIL

Copy code

image = Image.open(samples[i].path).convert('RGB')

Sorry, but I cannot give more of the code. I hope this can help

Fabio Grätz

03/02/2023, 6:07 PM

Does the first task return a single FlyteFile or a list of them?

Fabio Grätz

03/02/2023, 6:07 PM

(Since the 2nd task appears to get only a single one)

Victor Gustavo da Silva Oliveira

03/02/2023, 6:07 PM

A list, then I download each using a map_task

Fabio Grätz

03/02/2023, 6:07 PM

Got it

Fabio Grätz

03/02/2023, 6:08 PM

If you download the image in the second task, it won’t be available in the 3rd because it runs in its own container

Fabio Grätz

03/02/2023, 6:08 PM

You could omit the 2nd task and call

.download()

at the beginning of the 3rd task

Fabio Grätz

03/02/2023, 6:09 PM

Also, this caught my eye, not sure if it works:

return file.download()

Maybe replace with

Copy code

file.download()
return file

(The 3rd task will have to download again regardless of this though)

Victor Gustavo da Silva Oliveira

03/02/2023, 6:13 PM

So, each task must download the file? I can't see what is the advantage of using FlyteFile, though...

Fabio Grätz

03/02/2023, 6:14 PM

So, each task must download the file?

Yes, this is correct 🙂 Each task runs in its own pod in the kubernetes cluster which could potentially be on a different node than the previous task.

Fabio Grätz

03/02/2023, 6:15 PM

The benefit of the

FlyteFile

is that it takes care of the serialization and deserialization (to and from a bucket) when passing around files between tasks. Otherwise, you would have to take care of this (de-)serialization yourself.

Fabio Grätz

03/02/2023, 6:16 PM

Is the second task “doing a lot of work”? Otherwise I would recommend you omit the 2nd task and instead download the file directly in the 3rd task.

Victor Gustavo da Silva Oliveira

03/02/2023, 6:18 PM

It's not, but I intended to use the first ones as reference tasks in others workflows

Fabio Grätz

03/02/2023, 6:21 PM

Is the 2nd task doing something else than downloading the image?

Victor Gustavo da Silva Oliveira

03/02/2023, 6:22 PM

Nope. But we thoght that it will be better for our code if the latter tasks do not handle the download part, just deal with the manipulation

Fabio Grätz

03/02/2023, 6:26 PM

Ok understood. In Flyte, each task gets executed in a pod in kubernetes (think a container). These pods can run on completely different nodes (think virtual machines). This means that the tasks do not share a file system into which the 2nd task could download the image and the 3rd task will find it there. With other workflow orchestration engines, the user can only pass around strings between tasks. Flyte on the other hand allows you to pass more than strings between tasks, e.g. files. This is very convenient but behind the scenes all the inputs to a task have to be downloaded from a database/blob storage and all the return values have to be uploaded since the tasks run in different pods which can be on different virtual machines. So all the data that you want to manipulate in the 3rd task needs to be downloaded there. If the 2nd task does this …

Copy code

@task
def downooad_image(f: FlyteFile):
   f.download()
   return f

… then the image is first downloaded but then uploaded again behind the scenes when calling

return f

Victor Gustavo da Silva Oliveira

03/02/2023, 6:35 PM

Wow, that's great! But as you said, each task run in a separate pod, right? How come sometimes I get SIGKILL 9 from my cluster? I'm testing with just 6 small files, it should work fine. I was convinced that was the files downloaded previously in the second task...

Victor Gustavo da Silva Oliveira

03/02/2023, 6:36 PM

I was trying to rerun the workflow

Victor Gustavo da Silva Oliveira

03/02/2023, 6:36 PM

Maybe it's a question for another thread, but I believe there is a connection

Fabio Grätz

03/02/2023, 6:44 PM

You might get out-of-memory-killed.

Fabio Grätz

03/02/2023, 6:45 PM

Can you please try requesting more memory in the task that fails (see docs).

Victor Gustavo da Silva Oliveira

03/02/2023, 6:45 PM

That's what I thought, but I was afraid it was something related as the previous error. I will try to change resources. Thanks, Fabio!

Fabio Grätz

03/02/2023, 6:49 PM

You’re welcome 🙂

Fabio Grätz

03/02/2023, 6:49 PM

Feel free to ping me in a thread in case you have more problems.

Fabio Grätz

03/02/2023, 6:49 PM

Have a nice evening/day!

Victor Gustavo da Silva Oliveira

03/02/2023, 6:49 PM

Thanks! Same to you!

155 Views

Open in Slack

Previous Next