Hi teams Situation(probably): Context dead line exceeded when running

flytectl demo start

How to fix it? Environment:

ubuntu server 22.04
docker 26.1.4
python 3.10.12
flytectl 0.8.24

Description: I followed the tutorial in running locally workflow document. Following command stuck.

flytectl demo start

flyte-sandbox-64f5656c8f-kd66n encountered in to crash*.* Its log indicates that the crash reason seems to be like context dead line exceeded

{"json":{"src":"entrypoint.go:56"},"level":"info","msg":"Started propeller webhook","ts":"2024-06-08T00:43:17Z"}
panic: failed to initialize agent registry with error: failed to list agent: [&{dns:///flyteagent.flyte.svc.cluster.local:8000 true  map[gettask:10s] 10s}] with error: [rpc error: code = DeadlineExceeded desc = context deadline exceeded]

goroutine 32 [running]:
github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.newAgentPlugin()
        /flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:378 +0x45a
github.com/flyteorg/flyte/flyteplugins/go/tasks/plugins/webapi/agent.RegisterAgentPlugin()
        /flyteorg/build/flyteplugins/go/tasks/plugins/webapi/agent/plugin.go:403 +0xa5
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList.func1()
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xf
sync.(*Once).doSlow(0x0?, 0x1000100?)
        /usr/local/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.WranglePluginsAndGenerateFinalList({0x4302f80, 0xc00089ad70}, 0xc000773980, {0x7fc2084a7118, 0x5f3af40}, {0x43359c8?, 0xc00137f520})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/plugin_config.go:28 +0xac
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.(*Handler).Setup(0xc000903450, {0x4302f80, 0xc00089ad70}, {0x42f6930?, 0xc0009d2600})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/task/handler.go:231 +0x191
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/factory.(*handlerFactory).Setup(0xc000157b80, {0x4302f80, 0xc00089ad70}, {0x430f480?, 0xc000816320}, {0x42f6930, 0xc0009d2600})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/factory/handler_factory.go:73 +0x81f
github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).Initialize(0xc000816320, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/nodes/executor.go:464 +0xbf
github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).Initialize(0xc000482500, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/workflow/executor.go:385 +0x59
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Initialize(0x4302f80?, {0x4302f80?, 0xc00089ad70?})
        /flyteorg/build/flytepropeller/pkg/controller/handler.go:83 +0x26
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Initialize(...)
        /flyteorg/build/flytepropeller/pkg/controller/workers.go:121
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).run(0xc0005871f0, {0x4302f80, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:109 +0x82
github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Controller).Run(0xc0005871f0, {0x4302f80?, 0xc00089ad70})
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:96 +0x119
github.com/flyteorg/flyte/flytepropeller/pkg/controller.StartController({0x4302f80?, 0xc00089acd0?}, 0xc000830000, {0x343e1f4, 0x3}, {0x431e3c0?, 0xc00137e680}, 0xc001408000)
        /flyteorg/build/flytepropeller/pkg/controller/controller.go:612 +0x5b4
github.com/flyteorg/flyte/cmd/single.startPropeller.func2()
        /flyteorg/build/cmd/single/start.go:167 +0x94
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 82
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96

Thanks!

Hmm. Looks like the system is trying to talk to the agent service (which is usually another deployment) but it probably times out on the RPC call because the agent service isn’t up or it’s unresponsive. I’m not super familiar with the sandbox deployment but I can take a deeper look when I get to my computer.

try

flytectl demo start --disable-agent

Fix: https://github.com/flyteorg/flyte/pull/5459

Thanks @damp-lion-88352 The command works and i can do following tutorial!