<@USU6W5ATA> <@U05FFAF4FC2> I tested the PR and it...
# ask-the-communityg
Georgi Ivanov
11/24/2023, 4:31 PM@Kevin Su @L godlike I tested the PR and it works. I am not seeing the problem anymore.
Georgi Ivanov
11/24/2023, 4:32 PMsorry for replying outside of the thread, but slack does not give me an option to write in it. Anyway the fix looks good to me. Let me know once it’s been merged. Thanks for all your efforts!
L godlike
11/26/2023, 7:06 AM@Georgi Ivanov This is a new doc for databricks plugin, would you like to help use review it?
Thanks really much!
g
Georgi Ivanov
11/27/2023, 2:31 PMsure
Georgi Ivanov
11/27/2023, 2:34 PMis the source stored in github ?
Georgi Ivanov
11/27/2023, 2:35 PMthe source for the doc ?
l
L godlike
11/27/2023, 2:35 PMyes!
g
Georgi Ivanov
11/27/2023, 2:35 PMcan you give me a link please
l
L godlike
11/27/2023, 2:35 PML godlike
11/27/2023, 2:35 PMThank you very much
L godlike
11/27/2023, 2:36 PMIf you think there's any details need to be added for data scientist and ml engineer
please tell us, we will improve
g
Georgi Ivanov
11/27/2023, 2:37 PMyeah, i can only think of 1 thing
Georgi Ivanov
11/27/2023, 2:37 PMthe default container image if undefined when submitting a flyte workflow is flytekit (the release version matches the current version)
Georgi Ivanov
11/27/2023, 2:38 PMhowever, please correct me if I am wrong but the flytekit image by itself cannot be used as databricks container runner
l
L godlike
11/27/2023, 2:38 PMyes
L godlike
11/27/2023, 2:38 PMyou are right
L godlike
11/27/2023, 2:38 PMwe have to build by our own
L godlike
11/27/2023, 2:38 PMand this is important
g
Georgi Ivanov
11/27/2023, 2:38 PMwhat we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default image
Georgi Ivanov
11/27/2023, 2:39 PMas a minimum
Georgi Ivanov
11/27/2023, 2:39 PMthis is provided we don’t use image spec
Georgi Ivanov
11/27/2023, 2:39 PMand we want to have a “static” databricks runner
Georgi Ivanov
11/27/2023, 2:39 PMdid I miss it or it is already mentioned in the doc ?
l
L godlike
11/27/2023, 2:39 PMcan you talk about more
and we want to have a "static" databricks runnerL godlike
11/27/2023, 2:39 PMwhat we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default imageg
Georgi Ivanov
11/27/2023, 2:40 PMno, what I mean is that we need to have a runner and the default flytekit one cannot be used
l
L godlike
11/27/2023, 2:40 PMwe will add example in
<https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html>g
Georgi Ivanov
11/27/2023, 2:40 PMahum
Georgi Ivanov
11/27/2023, 2:40 PMok
l
L godlike
11/27/2023, 2:41 PMyes we need a runner for image spec
L godlike
11/27/2023, 2:41 PMenvd needs it
L godlike
11/27/2023, 2:42 PMok, we will improve the doc after this pr be merged
https://github.com/flyteorg/flytekit/pull/1951
L godlike
11/27/2023, 2:42 PMThank you really much
g
Georgi Ivanov
11/27/2023, 2:42 PMmaybe just add a link in the doc for this doc - https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html
l
L godlike
11/27/2023, 2:43 PMoh you mean the integration example
L godlike
11/27/2023, 2:43 PMno problem
g
Georgi Ivanov
11/27/2023, 2:43 PMyeah
Georgi Ivanov
11/27/2023, 2:43 PMbecause it is easily missed and a newcomer will struggle
l
L godlike
11/27/2023, 2:43 PMThis is helpful advice
L godlike
11/27/2023, 2:43 PMthank you very much
L godlike
11/27/2023, 2:43 PMreally much
g
Georgi Ivanov
11/27/2023, 2:43 PM👍
l
L godlike
11/27/2023, 2:43 PMadd it now
g
Georgi Ivanov
11/27/2023, 4:01 PMcan I ask a quick question?
Georgi Ivanov
11/27/2023, 4:02 PMI noticed this in the dockerfile - https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html
Georgi Ivanov
11/27/2023, 4:02 PM Copy code
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows and launch plans.
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tagGeorgi Ivanov
11/27/2023, 4:03 PMwhere does the tag argument comes from? If we build this image ourselves, how to we provide the FLYTE_INTERNAL_IMAGE in our CICD /
Georgi Ivanov
11/27/2023, 4:03 PM@L godlike
Georgi Ivanov
11/28/2023, 4:21 PM@Kevin Su can you help here ?
k
Kevin Su
11/28/2023, 7:26 PMThat env is optional. It’s not necessary to add that to the dockerfile.