Hi everybody I m working with FlyteFiles between two seppara Flyte #flyte-support

Hi everybody I'm working with FlyteFiles between t...

dry-ability-69144

03/02/2023, 5:42 PM

Hi everybody I'm working with FlyteFiles between two sepparate tasks: One download the file, the other works with it. In local environment, using sandbox, it work as expected. However, when running in a K8s cluster, it gives me this error:

Copy code

[Errno 2] No such file or directory: '/tmp/flytep4bezuvh/local_flytekit/d8e1f470ac5ed7ac3141ab1b6d83a854/6W2MGqmQ.jpg'

Can anybody help me with this?

cool-lifeguard-49380

03/02/2023, 5:57 PM

Is this in the task that uploaded the file or the one that downloaded it again?

cool-lifeguard-49380

03/02/2023, 5:58 PM

If in the latter, did you explicitly call

.download()

dry-ability-69144

03/02/2023, 5:59 PM

It's separated tasks In the first one I call

.download()

. Second one I'm just trying to use the

.path

to load the file. In this case, using PIL to load image

cool-lifeguard-49380

03/02/2023, 6:00 PM

Can you please try the following?

Copy code

@task
def upload_file() -> FlyteFile:
    # Save image to <some_path>
    return FlyteFile(path=<some_path>)

@task
def download_file(f: FlyteFile) -> None:
    f.download()

dry-ability-69144

03/02/2023, 6:01 PM

I forgot to mention, I'm getting the file from GCP. Let me share some snippets

dry-ability-69144

03/02/2023, 6:05 PM

I'm creating an list of FlyteFiles in the first task, using

Copy code

output.append(FlyteFile(path=f"gs://{bucket_name}/{img_prefix_path}"))

Then, in second task, I simply download it

Copy code

@task(cache=True, cache_version="1.0")
def load_file_as_flytefile(file: FlyteFile) -> FlyteFile:
    return file.download()

Then, finally, on the last task, I try to read it using PIL

Copy code

image = Image.open(samples[i].path).convert('RGB')

Sorry, but I cannot give more of the code. I hope this can help

cool-lifeguard-49380

03/02/2023, 6:07 PM

Does the first task return a single FlyteFile or a list of them?

cool-lifeguard-49380

03/02/2023, 6:07 PM

(Since the 2nd task appears to get only a single one)

dry-ability-69144

03/02/2023, 6:07 PM

A list, then I download each using a map_task

cool-lifeguard-49380

03/02/2023, 6:07 PM

Got it

cool-lifeguard-49380

03/02/2023, 6:08 PM

If you download the image in the second task, it won’t be available in the 3rd because it runs in its own container

cool-lifeguard-49380

03/02/2023, 6:08 PM

You could omit the 2nd task and call

.download()

at the beginning of the 3rd task

cool-lifeguard-49380

03/02/2023, 6:09 PM

Also, this caught my eye, not sure if it works:

return file.download()

Maybe replace with

Copy code

file.download()
return file

(The 3rd task will have to download again regardless of this though)

dry-ability-69144

03/02/2023, 6:13 PM

So, each task must download the file? I can't see what is the advantage of using FlyteFile, though...

cool-lifeguard-49380

03/02/2023, 6:14 PM

So, each task must download the file?

Yes, this is correct 🙂 Each task runs in its own pod in the kubernetes cluster which could potentially be on a different node than the previous task.

cool-lifeguard-49380

03/02/2023, 6:15 PM

The benefit of the

FlyteFile

is that it takes care of the serialization and deserialization (to and from a bucket) when passing around files between tasks. Otherwise, you would have to take care of this (de-)serialization yourself.

cool-lifeguard-49380

03/02/2023, 6:16 PM

Is the second task “doing a lot of work”? Otherwise I would recommend you omit the 2nd task and instead download the file directly in the 3rd task.

dry-ability-69144

03/02/2023, 6:18 PM

It's not, but I intended to use the first ones as reference tasks in others workflows

cool-lifeguard-49380

03/02/2023, 6:21 PM

Is the 2nd task doing something else than downloading the image?

dry-ability-69144

03/02/2023, 6:22 PM

Nope. But we thoght that it will be better for our code if the latter tasks do not handle the download part, just deal with the manipulation

cool-lifeguard-49380

03/02/2023, 6:26 PM

Ok understood. In Flyte, each task gets executed in a pod in kubernetes (think a container). These pods can run on completely different nodes (think virtual machines). This means that the tasks do not share a file system into which the 2nd task could download the image and the 3rd task will find it there. With other workflow orchestration engines, the user can only pass around strings between tasks. Flyte on the other hand allows you to pass more than strings between tasks, e.g. files. This is very convenient but behind the scenes all the inputs to a task have to be downloaded from a database/blob storage and all the return values have to be uploaded since the tasks run in different pods which can be on different virtual machines. So all the data that you want to manipulate in the 3rd task needs to be downloaded there. If the 2nd task does this …

Copy code

@task
def downooad_image(f: FlyteFile):
   f.download()
   return f

… then the image is first downloaded but then uploaded again behind the scenes when calling

return f

dry-ability-69144

03/02/2023, 6:35 PM

Wow, that's great! But as you said, each task run in a separate pod, right? How come sometimes I get SIGKILL 9 from my cluster? I'm testing with just 6 small files, it should work fine. I was convinced that was the files downloaded previously in the second task...

dry-ability-69144

03/02/2023, 6:36 PM

I was trying to rerun the workflow

dry-ability-69144

03/02/2023, 6:36 PM

Maybe it's a question for another thread, but I believe there is a connection

cool-lifeguard-49380

03/02/2023, 6:44 PM

You might get out-of-memory-killed.

cool-lifeguard-49380

03/02/2023, 6:45 PM

Can you please try requesting more memory in the task that fails (see docs).

dry-ability-69144

03/02/2023, 6:45 PM

That's what I thought, but I was afraid it was something related as the previous error. I will try to change resources. Thanks, Fabio!

cool-lifeguard-49380

03/02/2023, 6:49 PM

You’re welcome 🙂

🙌 1

cool-lifeguard-49380

03/02/2023, 6:49 PM

Feel free to ping me in a thread in case you have more problems.

cool-lifeguard-49380

03/02/2023, 6:49 PM

Have a nice evening/day!

dry-ability-69144

03/02/2023, 6:49 PM

Thanks! Same to you!

190 Views

Open in Slack

Previous Next