https://flyte.org logo
#ask-the-community
Title
# ask-the-community
r

Robert Ambrus

06/16/2023, 3:19 PM
Hi community! I'm trying to run a dummy python job on a 🧱 job cluster (_new_cluster_) using
pyflyte
like this:
Copy code
pyflyte run -i <corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1 --remote --destination-dir . dbx_example_job_cluster.py my_databricks_job
Job successfully submitted to Databricks, but the cluster creation failed because the IMAGE could not be pulled from our corporate docker registry (due to missing credentials). Analyzed the audit logs and found that the docker image config looks like this (_basic_auth_ block is missing):
Copy code
"docker_image": {
  "url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
}
I'm assuming that if we add
imagePullSecrets
to our service account as described here, Flyte will pass the Docker credentials to Databricks in job definition like this:
Copy code
"docker_image": {
  "url": "<corporate_docker_registry>/<prefix>/flyte-dbx-demo:0.0.1",
  "basic_auth": {
    "username": <user>,
    "password": <token>
  }
}
Can you please confirm? (Please note that I'm only experiencing this issue when trying to run a job on a _new_cluster,_ I was able to successfully complete a job on an _existing__cluster.) cc @Evan Sadler @Kevin Su
k

Kevin Su

06/16/2023, 7:36 PM
sorry, I didn’t know we can pass credentials to databricks job, so for now we won’t pass it to the dbx. I’ll add it. Contributions are welcome
r

Robert Ambrus

06/16/2023, 7:39 PM
All right, thanks for the quick feedback!
k

Kevin Su

06/16/2023, 7:42 PM
To workaround it, you can push image to the public registry, like docker hub.
r

Robert Ambrus

06/16/2023, 7:45 PM
That's not an option for us, we must use corporate docker registry.
k

Kevin Su

06/16/2023, 7:54 PM
I see, I’ll update it ASAP
e

Evan Sadler

06/16/2023, 8:07 PM
Thanks Kevin!
r

Robert Ambrus

06/20/2023, 1:42 PM
thank you @Kevin Su - I have seen this change: https://github.com/flyteorg/flyte/pull/3788 I presume we need to wait for the next release. Actually we are on Flyte
1.5.0
, any chance to get this change backported?
k

Kevin Su

06/21/2023, 6:27 AM
@Robert Ambrus could you help me test it? https://github.com/flyteorg/flyteplugins/pull/364. I’ve built an image.
pingsutw/flytepropeller:47f81950416273150c4547f6c9cb0caef61f0eec
. btw, you have to mount the imagePullSecrets to propeller first.
if you don’t want to mount the secret, you can also pass username and passport from flytekit
r

Robert Ambrus

06/21/2023, 11:15 AM
Ok, thanks @Kevin Su! I'm working on a POC to integrate Flyte with Databricks, the Flyte services are not managed by me, but let me discuss it with the team who is responsible for that.
a

Anantha Janakiraman

11/09/2023, 5:22 AM
Hi Kevin, Did the pull request above (364) for passing the image credentials to dbx during cluster creation get merged? I see that the pull request is still in draft mode in the old repo and did not find that change incorporated in the mono repo where all the flyteplugins code was moved after 10/09
12 Views