Hey everyone! I'm working to deploy Flyte in our GKE cluster via the standalone cluster documentation. We have requirements around leveraging Minio (already deployed on the cluster). I am able to get Flyte stood up and access it via console, however, when I "pyflyte run --remote" a workflow I get errors regarding connection to Minio based on the conneciton details I passed into the Flyte deployment via the helm chart. Is the pyflyte cli trying to connect directly to Minio and causing this error? I've tried adjusting the config.yaml with a storage config to point to the port-forwarded minio service but it still appears to try to connect to minio via teh config passed to the deployment.

Original exception: HTTPConnectionPool(host='signals-minio', port=9000): Max retries exceeded with url: /flyte/flytesnacks/development/YK2ZGX4HTNQPDYSEHLOIX7K2UM%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio_access_key%2F20230210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230210T152959Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11d5884c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

So pyflyte cli will upload the code using a signed url, it does not look like the host signals-minio is exposed to the internet

ah, that's great detail. We don't publicly expose the minio service. Are you able to point me in the direction of how to config FlyteAdmin?

Once I get to my computer or @jeev can you help

this will probably work if you are running

pyflyte run

from within the cluster

right, the specific scenario is local development and then pushing to our dev cluster to execute with our resources there

both flyte and

pyflyte run

need to be able to access minio at the same URL, if that makes sense. the signed url that flyteadmin returns to flytekit doesnt really make sense otherwise.

for other services using local dev we typically port-forward minio. Wondering if there is a way to override pyflyte's reference to minio use localhost instead of retrieving the minio host from the cluster

so flyteadmin retrieves a signed url from minio and passes that to flytekit. as far as flyteadmin is concerned, that url is accurate. it breaks down when the url leaves the cluster.

think i see why this path is used. Dependent on flyteadmin registering/declaring paths for flytekit to use locally

its a way of temporarily delegating writes to the bucket. so the client doesnt have to specifically have permissions set for that.

understood

if you feel up to it, you can hack flytekit code to do the address mapping to see if that works out for you. basically a string replace from the in-cluster url to localhost (assuming port-forwarding)

interesting. let me take a look