< glamorous carpet 83516> < damp lion 88352> I tested the PR Flyte #flyte-support

Join Slack

<@USU6W5ATA> <@U05FFAF4FC2> I tested the PR and it...

# flyte-support

billions-midnight-10687

11/24/2023, 4:31 PM

@glamorous-carpet-83516 @damp-lion-88352 I tested the PR and it works. I am not seeing the problem anymore.

👍 3

billions-midnight-10687

11/24/2023, 4:32 PM

sorry for replying outside of the thread, but slack does not give me an option to write in it. Anyway the fix looks good to me. Let me know once it’s been merged. Thanks for all your efforts!

❤️ 1

damp-lion-88352

11/26/2023, 7:05 AM

https://flyte--4445.org.readthedocs.build/en/4445/deployment/plugins/webapi/databricks.html#edit-the-iam-role-that-created-the-databricks-deployment

damp-lion-88352

11/26/2023, 7:06 AM

@billions-midnight-10687 This is a new doc for databricks plugin, would you like to help use review it? Thanks really much!

billions-midnight-10687

11/27/2023, 2:31 PM

sure

🙏 1

billions-midnight-10687

11/27/2023, 2:34 PM

is the source stored in github ?

billions-midnight-10687

11/27/2023, 2:35 PM

the source for the doc ?

damp-lion-88352

11/27/2023, 2:35 PM

yes!

billions-midnight-10687

11/27/2023, 2:35 PM

can you give me a link please

damp-lion-88352

11/27/2023, 2:35 PM

https://github.com/flyteorg/flyte/pull/4445

damp-lion-88352

11/27/2023, 2:35 PM

Thank you very much

damp-lion-88352

11/27/2023, 2:36 PM

If you think there's any details need to be added for data scientist and ml engineer please tell us, we will improve

billions-midnight-10687

11/27/2023, 2:37 PM

yeah, i can only think of 1 thing

billions-midnight-10687

11/27/2023, 2:37 PM

the default container image if undefined when submitting a flyte workflow is flytekit (the release version matches the current version)

billions-midnight-10687

11/27/2023, 2:38 PM

however, please correct me if I am wrong but the flytekit image by itself cannot be used as databricks container runner

damp-lion-88352

11/27/2023, 2:38 PM

yes

damp-lion-88352

11/27/2023, 2:38 PM

you are right

damp-lion-88352

11/27/2023, 2:38 PM

we have to build by our own

damp-lion-88352

11/27/2023, 2:38 PM

and this is important

billions-midnight-10687

11/27/2023, 2:38 PM

what we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default image

billions-midnight-10687

11/27/2023, 2:39 PM

as a minimum

billions-midnight-10687

11/27/2023, 2:39 PM

this is provided we don’t use image spec

billions-midnight-10687

11/27/2023, 2:39 PM

and we want to have a “static” databricks runner

billions-midnight-10687

11/27/2023, 2:39 PM

did I miss it or it is already mentioned in the doc ?

damp-lion-88352

11/27/2023, 2:39 PM

can you talk about more

and we want to have a "static" databricks runner

damp-lion-88352

11/27/2023, 2:39 PM

what we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default image

this is right!

billions-midnight-10687

11/27/2023, 2:40 PM

no, what I mean is that we need to have a runner and the default flytekit one cannot be used

damp-lion-88352

11/27/2023, 2:40 PM

we will add example in

<https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html>

here more

billions-midnight-10687

11/27/2023, 2:40 PM

ahum

billions-midnight-10687

11/27/2023, 2:40 PM

damp-lion-88352

11/27/2023, 2:41 PM

yes we need a runner for image spec

damp-lion-88352

11/27/2023, 2:41 PM

envd needs it

damp-lion-88352

11/27/2023, 2:42 PM

ok, we will improve the doc after this pr be merged https://github.com/flyteorg/flytekit/pull/1951

damp-lion-88352

11/27/2023, 2:42 PM

Thank you really much

billions-midnight-10687

11/27/2023, 2:42 PM

maybe just add a link in the doc for this doc - https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html

damp-lion-88352

11/27/2023, 2:43 PM

oh you mean the integration example

damp-lion-88352

11/27/2023, 2:43 PM

no problem

billions-midnight-10687

11/27/2023, 2:43 PM

yeah

billions-midnight-10687

11/27/2023, 2:43 PM

because it is easily missed and a newcomer will struggle

damp-lion-88352

11/27/2023, 2:43 PM

This is helpful advice

damp-lion-88352

11/27/2023, 2:43 PM

thank you very much

damp-lion-88352

11/27/2023, 2:43 PM

really much

billions-midnight-10687

11/27/2023, 2:43 PM

👍

🙏 1

damp-lion-88352

11/27/2023, 2:43 PM

add it now

gratitude thank you 1

🙇🏽 1

billions-midnight-10687

11/27/2023, 4:01 PM

can I ask a quick question?

billions-midnight-10687

11/27/2023, 4:02 PM

I noticed this in the dockerfile - https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html

billions-midnight-10687

11/27/2023, 4:02 PM

Copy code

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows and launch plans.
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

billions-midnight-10687

11/27/2023, 4:03 PM

where does the tag argument comes from? If we build this image ourselves, how to we provide the FLYTE_INTERNAL_IMAGE in our CICD /

billions-midnight-10687

11/27/2023, 4:03 PM

@damp-lion-88352

billions-midnight-10687

11/28/2023, 4:21 PM

@glamorous-carpet-83516 can you help here ?

glamorous-carpet-83516

11/28/2023, 7:26 PM

That env is optional. It’s not necessary to add that to the dockerfile.

Open in Slack

Previous Next