Robert Ambrus
06/16/2023, 3:19 PMpyflyte
like this:
pyflyte run -i <corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1 --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
Job successfully submitted to Databricks, but the cluster creation failed because the IMAGE could not be pulled from our corporate docker registry (due to missing credentials).
Analyzed the audit logs and found that the docker image config looks like this (_basic_auth_ block is missing):
"docker_image": {
"url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
}
I'm assuming that if we add imagePullSecrets
to our service account as described here, Flyte will pass the Docker credentials to Databricks in job definition like this:
"docker_image": {
"url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
"basic_auth": {
"username": <user>,
"password": <token>
}
}
Can you please confirm? (Please note that I'm only experiencing this issue when trying to run a job on a _new_cluster,_ I was able to successfully complete a job on an _existing__cluster.)
cc @Evan Sadler @Kevin SuKevin Su
06/16/2023, 7:36 PMRobert Ambrus
06/16/2023, 7:39 PMKevin Su
06/16/2023, 7:42 PMRobert Ambrus
06/16/2023, 7:45 PMKevin Su
06/16/2023, 7:54 PMEvan Sadler
06/16/2023, 8:07 PMRobert Ambrus
06/20/2023, 1:42 PM1.5.0
, any chance to get this change backported?Kevin Su
06/21/2023, 6:27 AMpingsutw/flytepropeller:47f81950416273150c4547f6c9cb0caef61f0eec
. btw, you have to mount the imagePullSecrets to propeller first.Robert Ambrus
06/21/2023, 11:15 AMAnantha Janakiraman
11/09/2023, 5:22 AM