aloof-painting-18735
06/16/2023, 3:19 PMpyflyte
like this:
pyflyte run -i <corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1 --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
Job successfully submitted to Databricks, but the cluster creation failed because the IMAGE could not be pulled from our corporate docker registry (due to missing credentials).
Analyzed the audit logs and found that the docker image config looks like this (_basic_auth_ block is missing):
"docker_image": {
"url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
}
I'm assuming that if we add imagePullSecrets
to our service account as described here, Flyte will pass the Docker credentials to Databricks in job definition like this:
"docker_image": {
"url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
"basic_auth": {
"username": <user>,
"password": <token>
}
}
Can you please confirm? (Please note that I'm only experiencing this issue when trying to run a job on a _new_cluster,_ I was able to successfully complete a job on an _existing__cluster.)
cc @billowy-winter-86593 @glamorous-carpet-83516glamorous-carpet-83516
06/16/2023, 7:36 PMaloof-painting-18735
06/16/2023, 7:39 PMglamorous-carpet-83516
06/16/2023, 7:42 PMaloof-painting-18735
06/16/2023, 7:45 PMglamorous-carpet-83516
06/16/2023, 7:54 PMbillowy-winter-86593
06/16/2023, 8:07 PMaloof-painting-18735
06/20/2023, 1:42 PM1.5.0
, any chance to get this change backported?glamorous-carpet-83516
06/21/2023, 6:27 AMpingsutw/flytepropeller:47f81950416273150c4547f6c9cb0caef61f0eec
. btw, you have to mount the imagePullSecrets to propeller first.glamorous-carpet-83516
06/21/2023, 6:28 AMaloof-painting-18735
06/21/2023, 11:15 AMfaint-piano-90133
11/09/2023, 5:22 AM