https://flyte.org logo
#ask-the-community
Title
# ask-the-community
g

Georgi Ivanov

11/24/2023, 4:31 PM
@Kevin Su @L godlike I tested the PR and it works. I am not seeing the problem anymore.
sorry for replying outside of the thread, but slack does not give me an option to write in it. Anyway the fix looks good to me. Let me know once it’s been merged. Thanks for all your efforts!
@Georgi Ivanov This is a new doc for databricks plugin, would you like to help use review it? Thanks really much!
g

Georgi Ivanov

11/27/2023, 2:31 PM
sure
is the source stored in github ?
the source for the doc ?
l

L godlike

11/27/2023, 2:35 PM
yes!
g

Georgi Ivanov

11/27/2023, 2:35 PM
can you give me a link please
l
Thank you very much
If you think there's any details need to be added for data scientist and ml engineer please tell us, we will improve
g

Georgi Ivanov

11/27/2023, 2:37 PM
yeah, i can only think of 1 thing
the default container image if undefined when submitting a flyte workflow is flytekit (the release version matches the current version)
however, please correct me if I am wrong but the flytekit image by itself cannot be used as databricks container runner
l

L godlike

11/27/2023, 2:38 PM
yes
you are right
we have to build by our own
and this is important
g

Georgi Ivanov

11/27/2023, 2:38 PM
what we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default image
as a minimum
this is provided we don’t use image spec
and we want to have a “static” databricks runner
did I miss it or it is already mentioned in the doc ?
l

L godlike

11/27/2023, 2:39 PM
can you talk about more
and we want to have a "static" databricks runner
what we need to add flytekit and flytekitplugins-spark python packages on top of the databricks default image
this is right!
g

Georgi Ivanov

11/27/2023, 2:40 PM
no, what I mean is that we need to have a runner and the default flytekit one cannot be used
l

L godlike

11/27/2023, 2:40 PM
we will add example in
<https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html>
here more
g

Georgi Ivanov

11/27/2023, 2:40 PM
ahum
ok
l

L godlike

11/27/2023, 2:41 PM
yes we need a runner for image spec
envd needs it
ok, we will improve the doc after this pr be merged https://github.com/flyteorg/flytekit/pull/1951
Thank you really much
g

Georgi Ivanov

11/27/2023, 2:42 PM
l

L godlike

11/27/2023, 2:43 PM
oh you mean the integration example
no problem
g

Georgi Ivanov

11/27/2023, 2:43 PM
yeah
because it is easily missed and a newcomer will struggle
l

L godlike

11/27/2023, 2:43 PM
This is helpful advice
thank you very much
really much
g

Georgi Ivanov

11/27/2023, 2:43 PM
👍
l

L godlike

11/27/2023, 2:43 PM
add it now
g

Georgi Ivanov

11/27/2023, 4:01 PM
can I ask a quick question?
Copy code
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows and launch plans.
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
where does the tag argument comes from? If we build this image ourselves, how to we provide the FLYTE_INTERNAL_IMAGE in our CICD /
@L godlike
@Kevin Su can you help here ?
k

Kevin Su

11/28/2023, 7:26 PM
That env is optional. It’s not necessary to add that to the dockerfile.