Hi all I have been trying Flyte for the past week or so and Flyte #flyte-support

Hi all. I have been trying Flyte for the past week...

sparse-addition-80036

06/21/2022, 5:31 PM

Hi all. I have been trying Flyte for the past week or so and I love it! I am new to ML Ops style tools and workflows, so I don't think I get all of the details yet. Question on the sandbox?: Is there a way to download the task outputs to the sandbox container? Right now my outputs for a task that creates a directory is this:

Copy code

{1 item
o0:{2 items
type:"multi-part blob"
uri:"<s3://my-s3-bucket/hi/ag5dp94bs4c846cn847s-n0-0/628c0053bbedf3f1db4d44362f48857e>"
}
}

Not sure where this multi-part blob resides

❤️ 3

acceptable-policeman-57188

06/21/2022, 7:38 PM

hi @sparse-addition-80036 what are you trying to do with the outputs? you can view them in the console or alternatively use the remote client for programmatic access

sparse-addition-80036

06/21/2022, 8:01 PM

Thanks @acceptable-policeman-57188! I am trying to check if the files have the right info in them. Can I use the remote client if I am running on a sandbox locally?

sparse-addition-80036

06/21/2022, 8:02 PM

And the ouputs of these tasks are of type FlyteFile, so only their paths show up on the console

acceptable-policeman-57188

06/21/2022, 8:44 PM

yup you can absolutely use flytekit remote with the local sandbox! just initialize the client with the path to your sandbox flytekit config (usually

~/.flyte/config.yaml

) and you should be good to go

acceptable-policeman-57188

06/21/2022, 8:44 PM

in fact i think the .auto() mode will pick this up for you

acceptable-policeman-57188

06/21/2022, 8:45 PM

i believe you can also use minio (the sandbox blob store) browser to browse the files too if you'd like

freezing-airport-6809

06/22/2022, 1:10 AM

So all intermediate outputs will reside in the configured bucket / s3/gcs or minio in sandbox case

👍 1

sparse-addition-80036

06/22/2022, 5:59 PM

Thank you, @acceptable-policeman-57188 and @freezing-airport-6809! I like the remote client. Giving it a shot now

👍 1

sparse-addition-80036

06/22/2022, 10:20 PM

So, I was able to figure out where the data is on the minio pod. Thanks for pointing me in that direction! The

remote_client.outputs

only shows the path to the files in the minio storage pod and I don't think it retrieves/downloads the files onto the local machine (that hosts the sandbox deployment). Is that correct? Is there a way to use flytekit or flytectl to interact with the minio storage files directly

freezing-airport-6809

06/22/2022, 10:22 PM

it should retrieve if you open - cc @thankful-minister-83577?

thankful-minister-83577

06/22/2022, 11:03 PM

(pinged on dm @sparse-addition-80036 but we can continue here too if you want)

163 Views

Open in Slack

Previous Next