Is it possible to use newer versions of spark with flyte? Seems like both the databricks and the spark operator paths bake in the assumption of using spark 3.3. Ideally I'd like to use the LTS version of databricks on a long running cluster but it appears that you have to use a custom image, is there any way to get around that?

Seems like both the databricks and the spark operator paths bake in the assumption of using spark 3.3

where did you see this

On the pyspark operator the image that gets automatically included uses 3.3. If I try to install a more recent version with pyspark==3.5.0, the 3.3.0 version baked into the flyte container gets resolved. If I try with databricks using flyte-agent I get a permissions error saying the user doesn't have permissions to navigate to

/root

you can use custom image for the spark task though. https://docs.flyte.org/en/latest/user_guide/customizing_dependencies/multiple_images_in_a_workflow.html

I tried that as well but when I did that I got an error about

/usr/bin/entrypoint.py

not existing.

@orange-lawyer-26259 could you share the ImageSpec config you're using?