Robert Ambrus
06/09/2023, 9:43 AMRobert Ambrus
06/09/2023, 12:14 PMRobert Ambrus
06/09/2023, 12:14 PM$ kubectl create secret generic -n flyte flyte-secret-auth
Robert Ambrus
06/09/2023, 12:14 PMjeev
jeev
Robert Ambrus
06/09/2023, 2:09 PMRobert Ambrus
06/09/2023, 2:11 PMRobert Ambrus
06/09/2023, 2:11 PMFLYTE_DATABRICKS_API_TOKEN
env variablejeev
Robert Ambrus
06/09/2023, 2:12 PMRobert Ambrus
06/09/2023, 2:16 PMsandbox
cluster (started by flytectl demo start), I don't have helm chart release that could be upgraded.
So this last does not seem to be compatible with the Sandbox
cluster. Is that correct?jeev
jeev
Robert Ambrus
06/09/2023, 2:21 PMjeev
Evan Sadler
06/09/2023, 9:27 PMDatabricks
config: flytekitplugins/spark/task.py#LL44C7-L44C17. It might work for testing purposes before @Kevin Su gets a chance to weigh in 🙂Evan Sadler
06/09/2023, 9:28 PMRobert Ambrus
06/12/2023, 8:39 AMRobert Ambrus
06/12/2023, 1:02 PMdatabricks_instance
alsoEvan Sadler
06/12/2023, 1:03 PMEvan Sadler
06/12/2023, 1:03 PMRobert Ambrus
06/12/2023, 1:03 PMEvan Sadler
06/12/2023, 1:06 PMRobert Ambrus
06/12/2023, 1:07 PMRobert Ambrus
06/13/2023, 6:09 PMDatabricks
config, but the run failed with:Robert Ambrus
06/13/2023, 6:09 PM{"json":{"exec_id":"f5a4876209ef64132940","ns":"flyte-user-flytesnacks-development","res_ver":"5168558","routine":"worker-38","wf":"flytesnacks:development:dbx_example.my_databricks_job"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [databricks]: secrets not found - file [/etc/secrets/FLYTE_DATABRICKS_API_TOKEN], Env [FLYTE_SECRET_FLYTE_DATABRICKS_API_TOKEN]].
Robert Ambrus
06/13/2023, 6:09 PMEvan Sadler
06/13/2023, 6:12 PMKevin Su
06/13/2023, 6:14 PMFLYTE_SECRET_FLYTE_DATABRICKS_API_TOKEN
Robert Ambrus
06/13/2023, 6:14 PMKevin Su
06/13/2023, 6:14 PMRobert Ambrus
06/13/2023, 6:15 PMRobert Ambrus
06/21/2023, 3:18 PMKevin Su
06/21/2023, 8:03 PMRobert Ambrus
06/22/2023, 7:21 AMRobert Ambrus
06/22/2023, 7:21 AMRobert Ambrus
06/22/2023, 7:39 AMRobert Ambrus
06/29/2023, 11:53 AMFlyte
to Databricks
, probably that would shed some light on the issue.
Is there an option for debug logging? Please note that I have already added the --verbose
flag to pyflyte
command, like this: pyflyte --verbose run -i <docker_registry>.<http://dkr.ecr.us-east-1.amazonaws.com/|dkr.ecr.us-east-1.amazonaws.com/><prefix>/<image>:<version> --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
👉 did not helpRobert Ambrus
06/29/2023, 1:37 PMRobert Ambrus
06/29/2023, 1:37 PMRobert Ambrus
06/29/2023, 1:37 PMRobert Ambrus
06/29/2023, 1:38 PM