Hi community! I'm trying to run a dummy python jo...
# ask-the-community
r
Hi community! I'm trying to run a dummy python job on a 🧱 job cluster (_new_cluster_) using
pyflyte
like this:
Copy code
pyflyte run -i <corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1 --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
Job successfully submitted to Databricks, but the cluster creation failed because the IMAGE could not be pulled from our corporate docker registry (due to missing credentials). Analyzed the audit logs and found that the docker image config looks like this (_basic_auth_ block is missing):
Copy code
"docker_image": {
  "url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
}
I'm assuming that if we add
imagePullSecrets
to our service account as described here, Flyte will pass the Docker credentials to Databricks in job definition like this:
Copy code
"docker_image": {
  "url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
  "basic_auth": {
    "username": <user>,
    "password": <token>
  }
}
Can you please confirm? (Please note that I'm only experiencing this issue when trying to run a job on a _new_cluster,_ I was able to successfully complete a job on an _existing__cluster.) cc @Evan Sadler @Kevin Su
k
sorry, I didn’t know we can pass credentials to databricks job, so for now we won’t pass it to the dbx. I’ll add it. Contributions are welcome
r
All right, thanks for the quick feedback!
k
To workaround it, you can push image to the public registry, like docker hub.
r
That's not an option for us, we must use corporate docker registry.
k
I see, I’ll update it ASAP
e
Thanks Kevin!
r
thank you @Kevin Su - I have seen this change: https://github.com/flyteorg/flyte/pull/3788 I presume we need to wait for the next release. Actually we are on Flyte
1.5.0
, any chance to get this change backported?
k
@Robert Ambrus could you help me test it? https://github.com/flyteorg/flyteplugins/pull/364. I’ve built an image.
pingsutw/flytepropeller:47f81950416273150c4547f6c9cb0caef61f0eec
. btw, you have to mount the imagePullSecrets to propeller first.
if you don’t want to mount the secret, you can also pass username and passport from flytekit
r
Ok, thanks @Kevin Su! I'm working on a POC to integrate Flyte with Databricks, the Flyte services are not managed by me, but let me discuss it with the team who is responsible for that.
a
Hi Kevin, Did the pull request above (364) for passing the image credentials to dbx during cluster creation get merged? I see that the pull request is still in draft mode in the old repo and did not find that change incorporated in the mono repo where all the flyteplugins code was moved after 10/09
139 Views