I think this deserves a separate thread

looking

I’m actually running into the same issue as stated here: My config for flyteagent:

plugin_config:
    plugins:
      # -- Agent service configuration for propeller.
      agent-service:
        # -- The default agent service to use for plugin tasks.
        defaultAgent:
          # -- The agent service endpoint propeller should connect to.
          endpoint: flyteagent.flyte.svc.cluster.local:8000
          # -- Whether the connection from propeller to the agent service should use TLS.
          insecure: true
        # -- The task types supported by the default agent.
        supportedTaskTypes:
        - sensor

i tried running a sensor workflow - the error is :

[1/1] currentAttempt done. Last Error: USER::failed to get grpc connection with error: failed to exit idle mode: passthrough: received empty target in Build()

my config map for propeller includes:

{{- if .Values.flyteagent.enabled }}
{{- with .Values.flyteagent.plugin_config }}
  agent_service.yaml: | {{ tpl (toYaml .) $ | nindent 4 }}
{{- end }}
{{- end }}

upon helm deploying my charts, i actually see propeller failing to start:

{"json":{"src":"handler.go:37"},"level":"debug","msg":"BranchNode::Setup: nothing to do","ts":"2024-05-28T23:33:57Z"}
{"json":{"src":"resourcemanager_util.go:22"},"level":"info","msg":"Using the NOOP resource manager","ts":"2024-05-28T23:33:57Z"}
panic: failed to initialize agent registry with error: failed to list agent: [&{flyteagent.flyte.svc.cluster.local:8000 true  map[] 10s}] with error: [rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup flyteagent.flyte.svc.cluster.local on 10.244.0.10:53: no such host"]

however , i can see that the flyteagent pod should be running fine:

(flytepython310) yaou@yaou-mn1 flyte-core % k logs flyteagent-84fd6c9c46-xnzrj
Starting up the server to expose the prometheus metrics...
Starting the agent service...
Starting Sensor that supports task categories ['sensor']
Starting Airflow Agent that supports task categories ['airflow']

any advice is appreciated 🙏

It's weird that I succeed in the sandbox

i think the sandbox is local and it would always work. What we are doing is spinning up the Flyte instance (propeller, admin, agent) brand new that isn’t sitting on local machines. We have all the other components working correctly, but having trouble introducing agent to our ecosystem.

Thank you for this feedback

hmm i see, any suggestions of fixes?

@gray-ram-51379 are you able to share entire propeller configmap with me

kubectl get cm flyte-propeller-config -o yaml > propeller-cm.yaml

@glamorous-carpet-83516 thank you, I sent you the paste bin of the config in a direct message

tyty.

So, I found that flyte helm chart hardcodes namespace

flyte

in the service addresses, so in case you have a different namespace, you need to set

endpoint: flyteagent.some-other-namespace-name.svc.cluster.local:8000

in two places:

flyteagent.plugin_config.plugins.agentService.defaultAgent.endpoint

and

configmap.enabled_plugins.plugins.agentService.defaultAgent.endpoint

so your default flyteagent is not running in the

flyte

namespace?

> so your default flyteagent is not running in the

flyte

namespace? yes, it's running in a namespace with a different name (and so are all the other Flyte services, btw)

does it work after you update the endpoint?

it didn’t work for me 😞 i redeployed it with new endpoint name (updated the namespace) and the error from propeller is:

{"json":{"src":"resourcemanager_util.go:22"},"level":"info","msg":"Using the NOOP resource manager","ts":"2024-05-29T18:40:45Z"}
panic: failed to initialize agent registry with error: failed to list agent: [&{flyteagent.flyte-dev.svc.cluster.local:8000 true  map[] 10s}] with error: [rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup flyteagent.flyte-dev.svc.cluster.local on 10.244.0.10:53: no such host"]

goroutine 546 [running]:
github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin()

is your agent running in the

flyte-dev

namespace

yeah, it is running in the

flyte-dev

namespace! I had a fix, i had to change the endpoint from

flyteagent.flyte-dev.svc.cluster.local:8000

, to just

flyteagent:8000

, it worked! no longer complaining about the connection issue!