Hey everyone! I'm working to deploy Flyte in our G...
# ask-the-community
t
Hey everyone! I'm working to deploy Flyte in our GKE cluster via the standalone cluster documentation. We have requirements around leveraging Minio (already deployed on the cluster). I am able to get Flyte stood up and access it via console, however, when I "pyflyte run --remote" a workflow I get errors regarding connection to Minio based on the conneciton details I passed into the Flyte deployment via the helm chart. Is the pyflyte cli trying to connect directly to Minio and causing this error? I've tried adjusting the config.yaml with a storage config to point to the port-forwarded minio service but it still appears to try to connect to minio via teh config passed to the deployment.
Copy code
Original exception: HTTPConnectionPool(host='signals-minio', port=9000): Max retries exceeded with url: /flyte/flytesnacks/development/YK2ZGX4HTNQPDYSEHLOIX7K2UM%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio_access_key%2F20230210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230210T152959Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11d5884c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
k
So pyflyte cli will upload the code using a signed url, it does not look like the host signals-minio is exposed to the internet
This is required for fast registration to work
Else it might be better to use just built docker containers
Which will mean - build container - push and then pyflyte register (non fast)
One other solution is to port forward minio service and change FlyteAdmin to tell flytekit to use local host. This is what we do with sandbox
t
ah, that's great detail. We don't publicly expose the minio service. Are you able to point me in the direction of how to config FlyteAdmin?
k
Once I get to my computer or @jeev can you help
j
this will probably work if you are running
pyflyte run
from within the cluster
but sounds like thats not the case.
t
right, the specific scenario is local development and then pushing to our dev cluster to execute with our resources there
j
both flyte and
pyflyte run
need to be able to access minio at the same URL, if that makes sense. the signed url that flyteadmin returns to flytekit doesnt really make sense otherwise.
the only way it will work is if there is address mapping somehow i believe. flytekit will need to map the signed url to something it can actually use.
sandbox does this in a VERY specific way.
t
for other services using local dev we typically port-forward minio. Wondering if there is a way to override pyflyte's reference to minio use localhost instead of retrieving the minio host from the cluster
j
so flyteadmin retrieves a signed url from minio and passes that to flytekit. as far as flyteadmin is concerned, that url is accurate. it breaks down when the url leaves the cluster.
that url might looks like http://signals-minio:9000/.....
so there needs to be some address mapping logic on either flyteadmin's or flytekit's side. the latter makes sense to me since the port-forwarding will be done client-side.
t
think i see why this path is used. Dependent on flyteadmin registering/declaring paths for flytekit to use locally
j
its a way of temporarily delegating writes to the bucket. so the client doesnt have to specifically have permissions set for that.
t
understood
j
if you feel up to it, you can hack flytekit code to do the address mapping to see if that works out for you. basically a string replace from the in-cluster url to localhost (assuming port-forwarding)
@Ketan (kumare3): do you know if flyteadmin already has configuration to do the address mapping?
@Taylor Stout im not 100% sure how/if this works, but here's an example config: https://github.com/flyteorg/flyteadmin/blob/ee8de0f4a7661b314ee3614f688163fc2936bffd/flyteadmin_config.yaml#L126
basically you want to set the endpoint override to
localhost
to a port that the client is expected to port-forward to.
t
interesting. let me take a look
232 Views