Hi all - I'm trying to use the <optuna plugin> for...
# flyte-support
m
Hi all - I'm trying to use the optuna plugin for flyte, and I'm having trouble running the example remotely. I'm running
flyte-binary
on K8s. I
pyflyte register
the eager task specified in the docs, and attempt to run it with
pyflyte --remote run
. I then hit the following error almost immediately:
Copy code
FlyteSystemUnavailableException: Flyte cluster is currently unavailable. Please make sure the cluster is up and running.
I've tried specifying the
remote
section of the eager task in a variety of ways but no dice. I'd really appreciate some help with this 🙏
above exception caused by
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: connect: Connection refused (111)"
    	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14, created_time:"2025-04-08T18:28:27.626869553+00:00"}"
    >
apparently the pod that gets spun up for the eager task is trying to talk to
localhost:30080
?
yeah no matter what I do I can't seem to set the remote endpoint correctly via
FlyteRemote
. It always keeps trying to use the default value.
h
What have you tried to set the config?
m
I've boiled down my problem to this snippet. I can't set the endpoint correctly in the
FlyteRemote
for eager workflows. I've tried the below code, along with various combinations of
FlyteRemote.auto
,
Config.auto
, specifying a config file instead of the endpoint, ... No matter what I do, it always tries to use the default endpoint of
localhost:30080
and fails since that isn't the right endpoint.
Copy code
from flytekit import eager
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, PlatformConfig
from flytekit import task

remote = FlyteRemote(
    # config=Config.for_endpoint(endpoint="dns:///calpha-service-flyte-grpc:8089", insecure=True),
    config=Config(
        platform=PlatformConfig(
            endpoint="dns:///calpha-service-flyte-grpc:8089", insecure=True, insecure_skip_verify=True
        )
    ),
    default_project="flytesnacks",
    default_domain="development",
)


@task
async def add_one(x: int) -> int:
    import time

    time.sleep(5)
    return x + 1


@eager(remote=remote)
async def my_test() -> str:
    y = await add_one(5)
    return "hello" + str(y)
Maybe there's some environment variable in the K8s deployment that's overriding this?
h
Copy code
@eager(remote=remote)
async def my_test() -> str:
    import os
    print (os.environ)
    y = await add_one(5)
    return "hello" + str(y)
Can you do this? I think you may be right...
m
Sure thing - one sec
nope - nothing in the environment that looks like a flyte endpoint. nothing like
30080
or
localhost
or anything that suggests the remote is being overrode. can't post all the envvars for security ofc
h
Do you see this in the logs?
Copy code
Constructing default remote with no config and
If you set
FLYTE_PLATFORM_URL
it should work
m
Thanks so much! I set the endpoint and now I'm hitting a different error, which means progress 😄 It's a DNS resolution failed now
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "DNS resolution failed for calpha-service-flyte-grpc:8089: C-ares status is not ARES_SUCCESS qtype=AAAA name=calpha-service-flyte-grpc is_balancer=0: Domain name not found"
    	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"DNS resolution failed for calpha-service-flyte-grpc:8089: C-ares status is not ARES_SUCCESS qtype=AAAA name=calpha-service-flyte-grpc is_balancer=0: Domain name not found", grpc_status:14, created_time:"2025-04-10T20:34:43.337184437+00:00"}"
    >
Setting the endpoint to the local kubernetes service endpoint got past the DNS resolution issue. Now it's complaining about SSL handshake failing; is there a way that I can tell flyte to allow insecure connection?
nvm chatgpt to the rescue!
Copy code
FLYTE_PLATFORM_INSECURE=true
lets me connect and use eager tasks. Thanks so much for all your help Haytham 🙏