We are setting up the databricks plugin and have it partiall Flyte #ask-the-community

We are setting up the databricks plugin and have i...

Evan Sadler

01/18/2023, 8:40 PM

We are setting up the databricks plugin and have it partially working! It works when the databricks PAT is passed into the task config:

Copy code

@task(
    task_config=Databricks(
        databricks_token="token"
))

For some reason it isn’t working with the default token, which is identical to the PAT that I pass in https://docs.flyte.org/en/latest/deployment/plugin_setup/webapi/databricks.html. Would love to get some help debugging or general advice on what to look for.

Kevin Su

01/18/2023, 8:46 PM

could you share your propeller config map? seems like some syntax format is incorrect.

Kevin Su

01/18/2023, 8:48 PM

sorry. my bad. the token should be set in the secret

Evan Sadler

01/18/2023, 8:48 PM

All good! We checked the secret and it looked okay to the naked eye.

Evan Sadler

01/18/2023, 8:48 PM

But obviously could be wrong

Kevin Su

01/18/2023, 8:48 PM

is the secret in the flyte namespace?

Evan Sadler

01/18/2023, 8:50 PM

yeah this found the secret:

Copy code

kubectl edit secret -n flyte flyte-secret-auth

Kevin Su

01/18/2023, 8:57 PM

I’m looking into it, and testing it. Maybe something wrong in the propeller

Evan Sadler

01/18/2023, 8:59 PM

Oh we are using the helm charts from 1.3.0b5

Evan Sadler

01/18/2023, 9:00 PM

maybe we need to upgrade to the release

Kevin Su

01/18/2023, 9:02 PM

if you don’t pass the token to the task config, what error you got in the propeller?

Evan Sadler

01/18/2023, 9:04 PM

I am not getting any errors from databricks even with the token passed in. I do see different messages printed. In the case where we don’t pass in the token:

Evan Sadler

01/18/2023, 9:04 PM

When we do pass in the token…we see “launched”

Kevin Su

01/18/2023, 9:04 PM

could you show me the log in propeller. kubectl logs flytepropeller -n flyte

Evan Sadler

01/18/2023, 9:07 PM

one sec

Evan Sadler

01/18/2023, 9:09 PM

Copy code

{"json":{"exec_id":"akg5bzqbx88fdt5798dr","ns":"flytesnacks-development","res_ver":"31741736","routine":"worker-0","wf":"flytesnacks:development:example.test_spark_wf"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [databricks]: Post \"<https://disc-staging-recs-workspace.cloud.databricks.com/api/2.0/jobs/runs/submit>\": net/http: invalid header field value \"Bearer u\\xaab\\xf7G\\x9e\\xdf\\u007fw\\xdbN\\x9c\\xed\\xfe=\\xeb\\xa6\\xdci\\xdd\\x1f\\xdf\\xcf;o\\xdf[\" for key Authorization]. Error Type[*errors.NodeErrorWithCause]","ts":"2023-01-18T21:07:29Z"}
E0118 21:07:29.208767       1 workers.go:102] error syncing 'flytesnacks-development/akg5bzqbx88fdt5798dr': failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [databricks]: Post "<https://disc-staging-recs-workspace.cloud.databricks.com/api/2.0/jobs/runs/submit>": net/http: invalid header field value "Bearer u\xaab\xf7G\x9e\xdf\u007fw\xdbN\x9c\xed\xfe=\xeb\xa6\xdci\xdd\x1f\xdf\xcf;o\xdf[" for key Authorization

Evan Sadler

01/18/2023, 9:12 PM

Then I see this

Evan Sadler

01/18/2023, 9:12 PM

Copy code

[]","ts":"2023-01-18T21:10:10Z"}
E0118 21:10:10.190687       1 workers.go:102] error syncing 'flytesnacks-development/amnnmh297qdr2gkwgnd8': Workflow[flytesnacks:development:example.test_spark_wf] failed. CausedByError: Failed to propagate Abort for workflow. Error: 0: panic when executing a plugin for TaskType [spark]. Stack: [goroutine 901 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort.func1.1()|github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort.func1.1()>
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:793 +0xea
panic({0x2289e60, 0xc00149b290})
	/usr/local/go/src/runtime/panic.go:838 +0x207
<http://github.com/flyteorg/flyteplugins/go/tasks/plugins/webapi/databricks.Plugin.Delete(|github.com/flyteorg/flyteplugins/go/tasks/plugins/webapi/databricks.Plugin.Delete(>{{0x2b7e1f0, 0xc0007e65d0}, 0xc000695440, {0x2b48d40, 0xc000b57f50}}, {0x2b6d998, 0xc00149aba0}, {0x2b5a1a0?, 0xc001abed80?})
	/go/pkg/mod/github.com/flyteorg/flyteplugins@v1.0.24/go/tasks/plugins/webapi/databricks/plugin.go:186 +0x2fa
<http://github.com/flyteorg/flyteplugins/go/tasks/pluginmachinery/internal/webapi.CorePlugin.Abort({{0x2645568|github.com/flyteorg/flyteplugins/go/tasks/pluginmachinery/internal/webapi.CorePlugin.Abort({{0x2645568>, 0xa}, {0x2b70e50, 0xc000b57f80}, {0x2b6e8b0, 0xc001828e40}, {{0x2b729e0, 0x3f89090}}, {{0x2b7e1f0, 0xc0007e6a50}, ...}}, ...)
	/go/pkg/mod/github.com/flyteorg/flyteplugins@v1.0.24/go/tasks/pluginmachinery/internal/webapi/core.go:111 +0x398
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort.func1(0xc002afa210|github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort.func1(0xc002afa210>?, {0x2b6d998, 0xc00149a0c0}, {0xc00299aa87?, 0xc001af8480?}, {0x2b6ff78, 0xc000bcd520}, 0x2b6f538?)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:800 +0x137
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort({{0x2b6f4f8|github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Abort({{0x2b6f4f8>, 0xc001882270}, {0x2b5a178, 0xc00169c200}, 0xc0012e8de0, 0xc0012e8e10, 0xc0012e8e40, {0x2b6f538, 0xc0012ef2c0}, 0xc0017ac580, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:802 +0x33d
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.Abort({{0x2b70f28|github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.Abort({{0x2b70f28>, 0xc001690340}, {{0xc000b8e280, {{...}, 0x0}, {0xc000ac8080, 0x4, 0x4}}, {0xc000b8e2a0, {{...}, ...}, ...}, ...}, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic/handler.go:260 +0x4d5
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).abort(0xc0014c6540|github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).abort(0xc0014c6540>?, {0x2b6d998, 0xc00149a0c0}, {0x2b6f338, 0xc0012ee000}, {0x2b7b810, 0xc001af8480}, {0x2652fc4, 0x11})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:456 +0xbe
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).AbortHandler(0xc0014c6540|github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).AbortHandler(0xc0014c6540>, {0x2b6d998, 0xc00149a060}, {0x2b7ddb8, 0xc000bb0f00}, {0x2b52518?, 0xc001c7e000}, {0x2b52540, 0xc001c7e000?}, {0x2b7b770, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:1073 +0x26a
<http://github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).AbortHandler(0xc0014c6540|github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).AbortHandler(0xc0014c6540>, {0x2b6d998, 0xc00149a060}, {0x2b7ddb8, 0xc000bb0f00}, {0x2b52518?, 0xc001c7e000}, {0x2b52540, 0xc001c7e000?}, {0x2b7b770, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:1124 +0xd65
<http://github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).cleanupRunningNodes(0xc00033bdc0|github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).cleanupRunningNodes(0xc00033bdc0>, {0x2b6d998, 0xc00149a060}, {0x2b7f440, 0xc001c7e000}, {0x2652fc4, 0x11})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:490 +0x24d
<http://github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).HandleAbortedWorkflow(0xc00033bdc0|github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).HandleAbortedWorkflow(0xc00033bdc0>, {0x2b6d998, 0xc00149a060}, 0xc001c7e000, 0x32)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:451 +0x237
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func1(0xc001b6d7e0|github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func1(0xc001b6d7e0>, {0x2b6d998?, 0xc00149a060?}, 0x2b3d2f0?, 0x2112b40?, 0x22b5950?)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:117 +0x93
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow(0xc001828000|github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow(0xc001828000>, {0x2b6d998, 0xc00181b980}, 0xc00288cf00)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:118 +0x2ba
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).Handle(0xc001828000|github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).Handle(0xc001828000>, {0x2b6d998, 0xc00181b980}, {0xc00288ae70, 0x17}, {0xc00288ae88, 0x14})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:258 +0xe4a
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem.func1(0xc000b72cf0|github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem.func1(0xc000b72cf0>, 0xc001b6df28, {0x2112b40?, 0xc0029d3790})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:88 +0x510
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem(0xc000b72cf0|github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem(0xc000b72cf0>, {0x2b6d998, 0xc00181b980})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:99 +0xf1
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).runWorker(0x2b6d998|github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).runWorker(0x2b6d998>?, {0x2b6d998, 0xc000b44000})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:115 +0xbd
<http://github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()|github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()>
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:150 +0x59
created by <http://github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run|github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run>
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:147 +0x285
]

Evan Sadler

01/18/2023, 9:18 PM

For when I use the PAT in the task config. The jobs are failing, but I don’t see any logs.

Evan Sadler

01/18/2023, 9:19 PM

Failing on databricks, but still running on Flyte

Kevin Su

01/18/2023, 9:19 PM

I saw the same error on my side. looking

Kevin Su

01/18/2023, 9:35 PM

@Evan Sadler One thing I forgot. the token in the secret should be based 64 encoded. you can encode the data through this website or use cli

Kevin Su

01/18/2023, 9:35 PM

https://kubernetes.io/docs/concepts/configuration/secret/#restriction-names-data

Kevin Su

01/18/2023, 9:35 PM

k8s will automatically decode it when reading it

Dan Rammer (hamersaw)

01/18/2023, 9:48 PM

@Kevin Su can we update the plugin so that it fails more gracefully? `panic`ing should never happen, rather this should fail the task with a useful error message.

Kevin Su

01/18/2023, 9:48 PM

yes, going to update it

Dan Rammer (hamersaw)

01/18/2023, 9:49 PM

and maybe update docs to reflect the base64 encoded secret

Evan Sadler

01/18/2023, 10:18 PM

@Kevin Su verified to work!

Evan Sadler

01/18/2023, 10:18 PM

THANK YOU

168 Views

Open in Slack

Previous Next