https://flyte.org
Join Slack
Hello! I dont have an agent running but I got this error which is crashing the pods: ```panic: faile...
# flyte-support
c
Hello! I dont have an agent running but I got this error which is crashing the pods:
Copy code
panic: failed to initialize agent registry with error: failed to list agent: [&{dns:///flyteagent.flyte.svc.cluster.local:8000 true map[gettask:10s] 10s}] with error: [rpc error: code = Unavailable desc = name resolver error: produced zero addresses]
<http://github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin|github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin> ( /flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:269 )
g
Are you using demo cluster?
c
no running my own in gcp
v1.8.0
h
It looks like you have agents enabled but not deployed?
c
Copy code
# flyteagent Configure Flyte Agent objects
flyteagent:
  # enable Flag to enable bundled Flyte Agent
  enable: false
g
looking, one sec
are you able to replace 
dns:///flyteagent.flyte.svc.cluster.local:8000
with 
""
in your helm chart
c
its not even set
not sure where its coming from
Copy code
co-pilot:
    # image Configure image to use for CoPilot sidecar
    image:
      # repository CoPilot sidecar image repository
      repository: <http://cr.flyte.org/flyteorg/flytecopilot|cr.flyte.org/flyteorg/flytecopilot> # FLYTECOPILOT_IMAGE
      # tag CoPilot sidecar image tag
      tag: v1.10.6 # FLYTECOPILOT_TAG
  # agentService Flyte Agent configuration
  agentService: {}
  #   Uncomment and modify to include configuration for Flyte Agent
  #   defaultGrpcEndpoint: agent-service.agent-namespace:8000
  #   supportedTaskTypes:
  #     - custom_task_type
only ref for 8000
g
panic: failed to initialize agent registry with
it’s in 1.10
c
i see
OK! Im on 1.10.6 now and have agent disabled still getting the same error
Copy code
# flyteagent Configure Flyte Agent objects
flyteagent:
  # enable Flag to enable bundled Flyte Agent
  enable: false
Copy code
co-pilot:
    # image Configure image to use for CoPilot sidecar
    image:
      # repository CoPilot sidecar image repository
      repository: <http://cr.flyte.org/flyteorg/flytecopilot|cr.flyte.org/flyteorg/flytecopilot> # FLYTECOPILOT_IMAGE
      # tag CoPilot sidecar image tag
      tag: v1.10.6 # FLYTECOPILOT_TAG
  # agentService Flyte Agent configuration
  agentService:
    defaultAgent:
      # endpoint: "dns:///flyteagent.flyte.svc.cluster.local:8000"
      insecure: true
      timeouts:
        GetTask: 10s
      defaultTimeout: 10s
    supportedTaskTypes:
      - default_task
  # externalConfigMap Specify an existing, external ConfigMap to use as configuration for Flyte
@thankful-minister-83577 could you help me figure out why this is happening?
Copy code
panic: failed to initialize agent registry with error: failed to list agent: [&{dns:///flyteagent.flyte.svc.cluster.local:8000 true  map[gettask:10s] 10s}] with error: [rpc error: code = Unavailable desc = name resolver error: produced zero addresses]

goroutine 46 [running]:
<http://github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin()|github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin()>
	/flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:269 +0x45a
<http://github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.RegisterAgentPlugin()|github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.RegisterAgentPlugin()>
	/flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:294 +0x9b
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList.func1()|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList.func1()>
	/flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xf
sync.(*Once).doSlow(0x0?, 0x3000106?)
	/usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList({0x466eee0|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList({0x466eee0>, 0xc000b41630}, 0xc000a8da80, {0x7f7aecd8f458, 0x626d6a0}, {0x46a0838?, 0xc0009aa000})
	/flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xac
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.(*Handler).Setup(0xc001090b60|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.(*Handler).Setup(0xc001090b60>, {0x466eee0, 0xc000b41630}, {0x46613f0?, 0xc0012f1578})
	/flyteorg/build/flytepropeller/pkg/controller/nodes/task/handler.go:231 +0x191
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/factory.(*handlerFactory).Setup(0xc0011b0000|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/factory.(*handlerFactory).Setup(0xc0011b0000>, {0x466eee0, 0xc000b41630}, {0x467adb0?, 0xc0012b4820}, {0x46613f0, 0xc0012f1578})
	/flyteorg/build/flytepropeller/pkg/controller/nodes/factory/handler_factory.go:73 +0x81f
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).Initialize(0xc0012b4820|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).Initialize(0xc0012b4820>, {0x466eee0, 0xc000b41630})
	/flyteorg/build/flytepropeller/pkg/controller/nodes/executor.go:454 +0xbf
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).Initialize(0xc000745f10|github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).Initialize(0xc000745f10>, {0x466eee0, 0xc000b41630})
	/flyteorg/build/flytepropeller/pkg/controller/workflow/executor.go:368 +0x59
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Initialize(0x466eee0|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Initialize(0x466eee0>?, {0x466eee0?, 0xc000b41630?})
	/flyteorg/build/flytepropeller/pkg/controller/handler.go:83 +0x26
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Initialize(...)|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Initialize(...)>
	/flyteorg/build/flytepropeller/pkg/controller/workers.go:121
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).run|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).run>(0xc00031b420, {0x466eee0, 0xc000b41630})
	/flyteorg/build/flytepropeller/pkg/controller/controller.go:108 +0x82
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).Run(0xc00031b420|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).Run(0xc00031b420>, {0x466eee0?, 0xc000b41630})
	/flyteorg/build/flytepropeller/pkg/controller/controller.go:95 +0x119
<http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.StartController(|github.com/flyteorg/flyte/flytepropeller/pkg/controller.StartController(>{0x466eee0?, 0xc000b41590?}, 0xc0009da000, {0x344bc6e, 0x3}, {0x4689800?, 0xc000346ea0}, 0xc000e7a000)
	/flyteorg/build/flytepropeller/pkg/controller/controller.go:605 +0x5b4
<http://github.com/flyteorg/flyte/cmd/single.startPropeller.func2()|github.com/flyteorg/flyte/cmd/single.startPropeller.func2()>
	/flyteorg/build/cmd/single/start.go:167 +0x94
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x56
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go> in goroutine 88
	/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x96
g
could you try this
Copy code
agentService:
    defaultAgent:
      endpoint: ""
      insecure: true
c
sure
I see that starting v1.9.2 flyteagent became a dep
and its not responding to enable false flag
g
you still see the agent deployment in the k8s when it equals to true?
c
No agent deployed
But the error still there
issue resolved
removing the string solved it!
thanks @glamorous-carpet-83516
g
we should add a condition here, sorry. updating it now
👍 1
c
will update to latest once resolved
@glamorous-carpet-83516 also the fact that binary image is set to latest causes problems https://github.com/flyteorg/flyte/blob/c049865cba017ad826405c7145cd3eccbc553232/charts/flyte-binary/values.yaml#L215
are there tags for this image?
g
^^
c
so this is latest meaning 1.10.7*?
g
c
sorry im not following which image in binary correspond to v1.10.6?
found what I was looking for
17 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by