krishna Yerramsetty

    krishna Yerramsetty

    3 months ago
    Hi all. I have been trying Flyte for the past week or so and I love it! I am new to ML Ops style tools and workflows, so I don't think I get all of the details yet. Question on the sandbox?: Is there a way to download the task outputs to the sandbox container? Right now my outputs for a task that creates a directory is this:
    {1 item
    o0:{2 items
    type:"multi-part blob"
    uri:"<s3://my-s3-bucket/hi/ag5dp94bs4c846cn847s-n0-0/628c0053bbedf3f1db4d44362f48857e>"
    }
    }
    Not sure where this multi-part blob resides
    k

    katrina

    3 months ago
    hi @krishna Yerramsetty what are you trying to do with the outputs? you can view them in the console or alternatively use the remote client for programmatic access
    krishna Yerramsetty

    krishna Yerramsetty

    3 months ago
    Thanks @katrina! I am trying to check if the files have the right info in them. Can I use the remote client if I am running on a sandbox locally?
    And the ouputs of these tasks are of type FlyteFile, so only their paths show up on the console
    k

    katrina

    3 months ago
    yup you can absolutely use flytekit remote with the local sandbox! just initialize the client with the path to your sandbox flytekit config (usually
    ~/.flyte/config.yaml
    ) and you should be good to go
    in fact i think the .auto() mode will pick this up for you
    i believe you can also use minio (the sandbox blob store) browser to browse the files too if you'd like
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    So all intermediate outputs will reside in the configured bucket / s3/gcs or minio in sandbox case
    krishna Yerramsetty

    krishna Yerramsetty

    3 months ago
    Thank you, @katrina and @Ketan (kumare3)! I like the remote client. Giving it a shot now
    So, I was able to figure out where the data is on the minio pod. Thanks for pointing me in that direction! The
    remote_client.outputs
    only shows the path to the files in the minio storage pod and I don't think it retrieves/downloads the files onto the local machine (that hosts the sandbox deployment). Is that correct? Is there a way to use flytekit or flytectl to interact with the minio storage files directly
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    it should retrieve if you open - cc @Yee?
    Yee

    Yee

    3 months ago
    (pinged on dm @krishna Yerramsetty but we can continue here too if you want)