Hey everyone I m working to deploy Flyte in our GKE cluster

Question

Hey everyone I m working to deploy Flyte in our GKE cluster via the standalone cluster documentation We have requirements around leveraging Minio already deployed on the cluster I am able to get Flyte stood up and access it via console however when I pyflyte run remote a workflow I get errors regarding connection to Minio based on the conneciton details I passed into the Flyte deployment via the helm chart Is the pyflyte cli trying to connect directly to Minio and causing this error I ve tried adjusting the config yaml with a storage config to point to the port forwarded minio service but it still appears to try to connect to minio via teh config passed to the deployment ```Original exception HTTPConnectionPool host= signals minio port=9000 Max retries exceeded with url flyte flytesnacks development YK2ZGX4HTNQPDYSEHLOIX7K2UM 3D 3D 3D 3D 3D 3D scriptmode tar gz X Amz Algorithm=AWS4 HMAC SHA256 amp X Amz Credential=minio access key 2F20230210 2Fus east 1 2Fs3 2Faws4 request amp X Amz Date=20230210T152959Z amp X Amz Expires=3600 amp X Amz SignedHeaders=content md5 3Bhost amp X Amz Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e Caused by NewConnectionError lt urllib3 connection HTTPConnection object at 0x11d5884c0 gt Failed to establish a new connection Errno 8 nodename nor servname provided or not known ```

Ketan (kumare3) · Answer

So pyflyte cli will upload the code using a signed url it does not look like the host signals minio is exposed to the internet

Ketan (kumare3) · Answer

This is required for fast registration to work

Ketan (kumare3) · Answer

Else it might be better to use just built docker containers

Ketan (kumare3) · Answer

Which will mean build container push and then pyflyte register non fast

Ketan (kumare3) · Answer

One other solution is to port forward minio service and change FlyteAdmin to tell flytekit to use local host This is what we do with sandbox

Taylor Stout · Answer

ah that s great detail We don t publicly expose the minio service Are you able to point me in the direction of how to config FlyteAdmin

Ketan (kumare3) · Answer

Once I get to my computer or < jeev> can you help

jeev · Answer

this will probably work if you are running `pyflyte run` from within the cluster

jeev · Answer

but sounds like thats not the case

Taylor Stout · Answer

right the specific scenario is local development and then pushing to our dev cluster to execute with our resources there

jeev · Answer

both flyte and `pyflyte run` need to be able to access minio at the same URL if that makes sense the signed url that flyteadmin returns to flytekit doesnt really make sense otherwise

jeev · Answer

the only way it will work is if there is address mapping somehow i believe flytekit will need to map the signed url to something it can actually use

jeev · Answer

sandbox does this in a VERY specific way

Taylor Stout · Answer

for other services using local dev we typically port forward minio Wondering if there is a way to override pyflyte s reference to minio use localhost instead of retrieving the minio host from the cluster

jeev · Answer

so flyteadmin retrieves a signed url from minio and passes that to flytekit as far as flyteadmin is concerned that url is accurate it breaks down when the url leaves the cluster

jeev · Answer

that url might looks like <http signals minio 9000 >

jeev · Answer

so there needs to be some address mapping logic on either flyteadmin s or flytekit s side the latter makes sense to me since the port forwarding will be done client side

Taylor Stout · Answer

think i see why this path is used Dependent on flyteadmin registering declaring paths for flytekit to use locally

jeev · Answer

its a way of temporarily delegating writes to the bucket so the client doesnt have to specifically have permissions set for that

Taylor Stout · Answer

understood

jeev · Answer

if you feel up to it you can hack flytekit code to do the address mapping to see if that works out for you basically a string replace from the in cluster url to localhost assuming port forwarding

jeev · Answer

< Ketan kumare3 > do you know if flyteadmin already has configuration to do the address mapping

jeev · Answer

ah this looks like it used

jeev · Answer

< Taylor Stout> im not 100 sure how if this works but here s an example config

jeev · Answer

basically you want to set the endpoint override to `localhost` to a port that the client is expected to port forward to

Taylor Stout · Answer

interesting let me take a look