I have a very odd issue. I've configured my flyte client (flytectl) such that it points to my flyte-core deployment's admin/console ingresses. My goal is to not have to do any port-forwarding for minio nor flyteadmin. After configuring my console/admin http and grpc ingresses, I run

flytectl get projects

and successfully get my projects in my cluster

--------------- --------------- --------------------------- 
| ID (4)        | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| #####        | #####        | #############             |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
4 rows

I can also reach my minio instance in the cluster without port-forwarding. However, when I try to run python code on the remote server using

pyflyte -v run --remote -p ###### -d development  some_example.py some_workflow

, I get the following error

...
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "DNS resolution failed for flyte.somedomain.com: C-ares status is not ARES_SUCCESS qtype=A 
name=flyte.somedomain.com is_balancer=0: Timeout while contacting DNS servers"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"DNS resolution failed for flyte.somedomain.com: 
C-ares status is not ARES_SUCCESS qtype=A name=flyte.somedomain.com is_balancer=0: Timeout while contacting DNS servers", 
grpc_status:14, created_time:"2025-02-20T15:43:24.043117-08:00"}"
...
FlyteSystemUnavailableException: Flyte cluster is currently unavailable. Please make sure the cluster is up and running.
...
RuntimeError: Failed to get signed url for fastec41b71eb12116e44b8039fa355d3577.tar.gz.

All the pods are running normally, and when configuring flyte client to use port-forwarded address, everything works fine. I also checked to see if I could find my configured hostname from a test pod running in the same cluster (running

host <http://flyte.somedomain.com|flyte.somedomain.com>

) and I get back the flyteadmin's pod IP, so it doesn't seem to be an issue with an internal cluster DNS lookup Any ideas as to why flytectl can talk to the flyteadmin pod running in the cluster, but pyflyte fails??