Robert Ambrus
06/09/2023, 9:43 AM$ kubectl create secret generic -n flyte flyte-secret-auth
jeev
Robert Ambrus
06/09/2023, 2:09 PMFLYTE_DATABRICKS_API_TOKEN
env variablejeev
Robert Ambrus
06/09/2023, 2:12 PMsandbox
cluster (started by flytectl demo start), I don't have helm chart release that could be upgraded.
So this last does not seem to be compatible with the Sandbox
cluster. Is that correct?jeev
Robert Ambrus
06/09/2023, 2:21 PMjeev
Evan Sadler
06/09/2023, 9:27 PMDatabricks
config: flytekitplugins/spark/task.py#LL44C7-L44C17. It might work for testing purposes before @Kevin Su gets a chance to weigh in 🙂Robert Ambrus
06/12/2023, 8:39 AMdatabricks_instance
alsoEvan Sadler
06/12/2023, 1:03 PMRobert Ambrus
06/12/2023, 1:03 PMEvan Sadler
06/12/2023, 1:06 PMRobert Ambrus
06/12/2023, 1:07 PMDatabricks
config, but the run failed with:{"json":{"exec_id":"f5a4876209ef64132940","ns":"flyte-user-flytesnacks-development","res_ver":"5168558","routine":"worker-38","wf":"flytesnacks:development:dbx_example.my_databricks_job"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [databricks]: secrets not found - file [/etc/secrets/FLYTE_DATABRICKS_API_TOKEN], Env [FLYTE_SECRET_FLYTE_DATABRICKS_API_TOKEN]].
Evan Sadler
06/13/2023, 6:12 PMKevin Su
06/13/2023, 6:14 PMFLYTE_SECRET_FLYTE_DATABRICKS_API_TOKEN
Robert Ambrus
06/13/2023, 6:14 PMKevin Su
06/13/2023, 6:14 PMRobert Ambrus
06/13/2023, 6:15 PMKevin Su
06/21/2023, 8:03 PMRobert Ambrus
06/22/2023, 7:21 AMFlyte
to Databricks
, probably that would shed some light on the issue.
Is there an option for debug logging? Please note that I have already added the --verbose
flag to pyflyte
command, like this: pyflyte --verbose run -i <docker_registry>.<http://dkr.ecr.us-east-1.amazonaws.com/|dkr.ecr.us-east-1.amazonaws.com/><prefix>/<image>:<version> --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
👉 did not help