Hi all I have been trying Flyte for the past week or so and Flyte #ask-the-community

Hi all. I have been trying Flyte for the past week...

krishna Yerramsetty

06/21/2022, 5:31 PM

Hi all. I have been trying Flyte for the past week or so and I love it! I am new to ML Ops style tools and workflows, so I don't think I get all of the details yet. Question on the sandbox?: Is there a way to download the task outputs to the sandbox container? Right now my outputs for a task that creates a directory is this:

Copy code

{1 item
o0:{2 items
type:"multi-part blob"
uri:"<s3://my-s3-bucket/hi/ag5dp94bs4c846cn847s-n0-0/628c0053bbedf3f1db4d44362f48857e>"
}
}

Not sure where this multi-part blob resides

❤️ 3

katrina

06/21/2022, 7:38 PM

hi @krishna Yerramsetty what are you trying to do with the outputs? you can view them in the console or alternatively use the remote client for programmatic access

krishna Yerramsetty

06/21/2022, 8:01 PM

Thanks @katrina! I am trying to check if the files have the right info in them. Can I use the remote client if I am running on a sandbox locally?

krishna Yerramsetty

06/21/2022, 8:02 PM

And the ouputs of these tasks are of type FlyteFile, so only their paths show up on the console

katrina

06/21/2022, 8:44 PM

yup you can absolutely use flytekit remote with the local sandbox! just initialize the client with the path to your sandbox flytekit config (usually

~/.flyte/config.yaml

) and you should be good to go

katrina

06/21/2022, 8:44 PM

in fact i think the .auto() mode will pick this up for you

katrina

06/21/2022, 8:45 PM

i believe you can also use minio (the sandbox blob store) browser to browse the files too if you'd like

Ketan (kumare3)

06/22/2022, 1:10 AM

So all intermediate outputs will reside in the configured bucket / s3/gcs or minio in sandbox case

👍 1

krishna Yerramsetty

06/22/2022, 5:59 PM

Thank you, @katrina and @Ketan (kumare3)! I like the remote client. Giving it a shot now

👍 1

krishna Yerramsetty

06/22/2022, 10:20 PM

So, I was able to figure out where the data is on the minio pod. Thanks for pointing me in that direction! The

remote_client.outputs

only shows the path to the files in the minio storage pod and I don't think it retrieves/downloads the files onto the local machine (that hosts the sandbox deployment). Is that correct? Is there a way to use flytekit or flytectl to interact with the minio storage files directly

Ketan (kumare3)

06/22/2022, 10:22 PM

it should retrieve if you open - cc @Yee?

Yee

06/22/2022, 11:03 PM

(pinged on dm @krishna Yerramsetty but we can continue here too if you want)

162 Views

Open in Slack

Previous Next