https://flyte.org logo
Join Slack
Powered by
:thread: Flyte Propeller Stacktrace
# flyte-support
c
🧵 Flyte Propeller Stacktrace
Copy code
time="2024-06-04T23:26:51Z" level=info msg=------------------------------------------------------------------------
time="2024-06-04T23:26:51Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2024-06-04 23:26:50.913052211 +0000 UTC m=+0.001638714]"
time="2024-06-04T23:26:51Z" level=info msg=------------------------------------------------------------------------
time="2024-06-04T23:26:51Z" level=info msg="Detected: 32 CPU's\n"
I0604 23:26:53.813337       1 leaderelection.go:250] attempting to acquire leader lease flyte-data-plane/propeller-leader...
I0604 23:27:09.489740       1 leaderelection.go:260] successfully acquired lease flyte-data-plane/propeller-leader
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
        >  goroutine 769 [running]:
        >  runtime/debug.Stack()
        >       /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
        >  <http://sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()|sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()>
        >       /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/log.go:60 +0xcd
        >  <http://sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc00091d600|sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc00091d600>, 0x1600000000000000?)
        >       /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/deleg.go:111 +0x32
        >  <http://github.com/go-logr/logr.Logger.Info({{0x2f582b0|github.com/go-logr/logr.Logger.Info({{0x2f582b0>?, 0xc00091d600?}, 0xc002628000?}, {0xc002628000, 0x269}, {0x0, 0x0, 0x0})
        >       /go/pkg/mod/github.com/go-logr/logr@v1.3.0/logr.go:280 +0x72
        >  <http://sigs.k8s.io/controller-runtime/pkg/log.(*KubeAPIWarningLogger).HandleWarningHeader(0xc000b65200|sigs.k8s.io/controller-runtime/pkg/log.(*KubeAPIWarningLogger).HandleWarningHeader(0xc000b65200>?, 0x1?, {0x2987392?, 0x7?}, {0xc002628000?, 0x269?})
        >       /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/warning_handler.go:65 +0x167
        >  <http://k8s.io/client-go/rest.handleWarnings(0xc001f4bd40|k8s.io/client-go/rest.handleWarnings(0xc001f4bd40>?, {0x2f267a0?, 0xc000b65200?})
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/warnings.go:144 +0xef
        >  <http://k8s.io/client-go/rest.(*Request).transformResponse(0xc002682c60|k8s.io/client-go/rest.(*Request).transformResponse(0xc002682c60>, 0xc002266c60, 0xc002225f00)
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1179 +0x5c8
        >  <http://k8s.io/client-go/rest.(*Request).Do.func1(0xc0025c59c0|k8s.io/client-go/rest.(*Request).Do.func1(0xc0025c59c0>?, 0x0?)
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1064 +0x31
        >  <http://k8s.io/client-go/rest.(*Request).request.func3.1(...)|k8s.io/client-go/rest.(*Request).request.func3.1(...)>
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1039
        >  <http://k8s.io/client-go/rest.(*Request).request.func3|k8s.io/client-go/rest.(*Request).request.func3>(0xc002266c60, 0xc0025f17c0, {0x2f52a48?, 0xc0025c59c0?}, 0x0?, 0x0?, 0xc00225d980?, {0x0?, 0x0?}, 0x2ae1c00)
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1046 +0xd7
        >  <http://k8s.io/client-go/rest.(*Request).request(0xc002682c60|k8s.io/client-go/rest.(*Request).request(0xc002682c60>, {0x2f521b8, 0xc002595030}, 0x2?)
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1048 +0x4ed
        >  <http://k8s.io/client-go/rest.(*Request).Do(0xc002682c60|k8s.io/client-go/rest.(*Request).Do(0xc002682c60>, {0x2f52110, 0xc0025c9f50})
        >       /go/pkg/mod/k8s.io/client-go@v0.28.2/rest/request.go:1063 +0xb0
        >  <http://sigs.k8s.io/controller-runtime/pkg/client.(*typedClient).Create(0xc00085cf30|sigs.k8s.io/controller-runtime/pkg/client.(*typedClient).Create(0xc00085cf30>, {0x2f52110, 0xc0025c9f50}, {0x2f73260?, 0xc002686480}, {0x0, 0x0, 0x52?})
        >       /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/typed_client.go:48 +0x346
        >  <http://sigs.k8s.io/controller-runtime/pkg/client.(*client).Create(0xc00085cf30|sigs.k8s.io/controller-runtime/pkg/client.(*client).Create(0xc00085cf30>?, {0x2f52110?, 0xc0025c9f50?}, {0x2f73260?, 0xc002686480?}, {0x0?, 0x0?, 0x0?})
        >       /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/client.go:288 +0xe5
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/executors.flyteK8sClient.Create({{0x2f63c60|github.com/flyteorg/flyte/flytepropeller/pkg/controller/executors.flyteK8sClient.Create({{0x2f63c60>, 0xc00085cf30}, {0x7f08bf7d0ff8, 0xc000e808c0}, {0x2f33ca8, 0xc00091d880}}, {0x2f52110, 0xc0025c9f50}, {0x2f73260, 0xc002686480}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/executors/kube.go:88 +0xfb
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.(*PluginManager).launchResource.func1()|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.(*PluginManager).launchResource.func1()>
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go:214 +0x6d
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/backoff.(*ComputeResourceAwareBackOffHandler).Handle(0xc001f4f030|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/backoff.(*ComputeResourceAwareBackOffHandler).Handle(0xc001f4f030>, {0x2f52110, 0xc0025c9f50}, 0xc0025f21a0, 0xc0025e4e10)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/backoff/handler.go:147 +0x294
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.(*PluginManager).launchResource(0xc0025f2910|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.(*PluginManager).launchResource(0xc0025f2910>, {0x2f52110?, 0xc0025c9f50}, {0x2f63488, 0xc002279930})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go:213 +0x551
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.PluginManager.Handle({{0x298a6d3|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task/k8s.PluginManager.Handle({{0x298a6d3>, 0x9}, {0x2f52428, 0x4589600}, {0x2f33028, 0xc0004d4900}, {0x7f08bf44ac38, 0xc0007429c0}, {{0x2f6df18, 0xc000debfa0}, ...}, ...}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go:332 +0x11d
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin.func1(0x298a700|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin.func1(0x298a700>?, {0x2f52110?, 0xc0025c9d70}, {0x2f56080?, 0xc000591e40}, 0x44b6760?)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:385 +0x174
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin({{0x2f55000|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin({{0x2f55000>, 0xc000a16ac8}, {0x2f3b4f0, 0xc000a042c0}, 0xc001093f50, 0xc001093f80, 0xc001093fb0, {0x2f56080, 0xc000591e40}, 0xc000129dc0, ...}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:387 +0x8e
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.Handle({{0x2f55000|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/task.Handler.Handle({{0x2f55000>, 0xc000a16ac8}, {0x2f3b4f0, 0xc000a042c0}, 0xc001093f50, 0xc001093f80, 0xc001093fb0, {0x2f56080, 0xc000591e40}, 0xc000129dc0, ...}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:571 +0x865
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.handleParentNode({{0x2f5ccd8|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.handleParentNode({{0x2f5ccd8>, 0xc0006fd860}, {{0xc000ddc350, {{...}, 0x0}, {0xc00091c080, 0x4, 0x4}}, {0xc000ddc370, {{...}, ...}, ...}, ...}, ...}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic/handler.go:64 +0xdc
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.Handle({{0x2f5ccd8|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.Handle({{0x2f5ccd8>, 0xc0006fd860}, {{0xc000ddc350, {{...}, 0x0}, {0xc00091c080, 0x4, 0x4}}, {0xc000ddc370, {{...}, ...}, ...}, ...}, ...}, ...)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic/handler.go:221 +0xb65
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).execute(0xc000400c00|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).execute(0xc000400c00>, {0x2f52110?, 0xc0025c99e0}, {0x2f550c0, 0xc000263400}, {0x2f66a98, 0xc001bed5c0}, {0x2f763d8, 0xc001b99dc0})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:828 +0x147
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleQueuedOrRunningNode(0xc000400c00|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleQueuedOrRunningNode(0xc000400c00>, {0x2f52110?, 0xc0025c99e0}, {0x2f66a98, 0xc001bed5c0}, {0x2f550c0, 0xc000263400})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:1122 +0xa9d
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).HandleNode(0xc000400c00|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*nodeExecutor).HandleNode(0xc000400c00>, {0x2f52110?, 0xc0025c9770?}, {0x2f361f8, 0xc0025de000}, {0x2f66a98, 0xc001bed5c0}, {0x2f550c0, 0xc000263400?})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:1397 +0x3b8
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).RecursiveNodeHandler(0xc0005187d0|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).RecursiveNodeHandler(0xc0005187d0>, {0x2f52110, 0xc0025c9410}, {0x2f703e0, 0xc0025d6370}, {0x2f361f8, 0xc0025de000}, {0x2f52dc8?, 0xc0025de000?}, {0x2f69e70, ...})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:234 +0x91c
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).handleDownstream(0x29d2d68|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).handleDownstream(0x29d2d68>?, {0x2f52110, 0xc0025c9410}, {0x2f703e0, 0xc0025d6370}, {0x2f361f8, 0xc0025de000?}, {0x2f52dc8?, 0xc0025de000}, {0x2f69e70, ...})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:300 +0x47a
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).RecursiveNodeHandler(0xc0005187d0|github.com/flyteorg/flyte/flytepropeller/pkg/controller/nodes.(*recursiveNodeExecutor).RecursiveNodeHandler(0xc0005187d0>, {0x2f52110, 0xc0025c9410}, {0x2f703e0, 0xc0025d6370}, {0x2f361f8, 0xc0025de000}, {0x2f52dc8?, 0xc0025de000?}, {0x2f69e70, ...})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:241 +0xb39
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).handleRunningWorkflow(0xc001062c80|github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).handleRunningWorkflow(0xc001062c80>, {0x2f52110, 0xc0025c9410}, 0xc0025de000)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:161 +0x1bb
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).HandleFlyteWorkflow(0xc001062c80|github.com/flyteorg/flyte/flytepropeller/pkg/controller/workflow.(*workflowExecutor).HandleFlyteWorkflow(0xc001062c80>, {0x2f52110?, 0xc0025c9410}, 0xc0025de000)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:418 +0x3d8
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func2(0xc000277940|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func2(0xc000277940>, {0x2f52110?, 0xc0025c9410}, 0xc0025f76d0, 0x2346d80?)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:142 +0x178
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow(0xc000277940|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow(0xc000277940>, {0x2f52110, 0xc0025c9140}, 0xc002476a00)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:143 +0x493
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).streak(0xc000277940|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).streak(0xc000277940>, {0x2f52110?, 0xc00223c750?}, 0xc002476a00, 0x0)
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:298 +0x2a8
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Handle(0xc000277940|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*Propeller).Handle(0xc000277940>, {0x2f52110?, 0xc00223c6f0?}, {0xc0020f1110, 0x17}, {0xc0020f1128, 0x14})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:241 +0xa2b
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem.func1(0xc0011302d0|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem.func1(0xc0011302d0>, 0xc0025f7f20, {0x2346d80?, 0xc0022440b0})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:89 +0x503
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem(0xc0011302d0|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem(0xc0011302d0>, {0x2f52110, 0xc00223c6f0})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:100 +0xdd
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).runWorker(0x2f52110|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).runWorker(0x2f52110>?, {0x2f52110?, 0xc001283320})
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:116 +0xb1
        >  <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()>
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:151 +0x4f
        >  created by <http://github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run|github.com/flyteorg/flyte/flytepropeller/pkg/controller.(*WorkerPool).Run> in goroutine 460
        >       /go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:148 +0x285
Wanted to see if anyone has seen this before I file an issue. Seems to get triggered the first time I try to run a task. This is on 
<http://cr.flyte.org/flyteorg/flytepropeller:v1.12.0-b0|cr.flyte.org/flyteorg/flytepropeller:v1.12.0-b0>
t
i have not seen this before sorry
can you describe the set-up more?
are you able to run any tasks at all? what was the last known good configuration and what kind of deployment is this? (flyte-binary helm chart vs core, aks, eks, etc)
c
Its the core helm chart. Tasks seem to run successfully even with the stacktrace.
t
for all tasks or just some tasks?
c
I have only tried Python Tasks and they seem to run
t
would you mind putting this into a GH issue? i’ll ask around internally some more
if you could share the propeller config map also that might be helpful, redacted if there’s anything sensitive in there.
c
Yeah I'll file an issue what what details I can share
t
thanks
f
It seems you have some crd not installed maybe, but you are using line spark, distributed training etc
13 Views
Open in Slack
PreviousNext
Powered by