@ancient-wolf-19325 Is it possible to run spark java or scala jobs as part of a python workflow defined using flytekit spark tasks? How does that work?

Yes, it is possible to run Spark jobs written in Java or Scala as part of a Python workflow defined using Flytekit Spark tasks. This can be achieved by configuring the Spark task to include the necessary JAR files and class paths for the Java/Scala code. According to Flyte discussion, you can include JARs built from your Scala code in the container image used for the Spark task. You can then set the

spark.driver.extraClassPath

and

spark.executor.extraClassPath

in the task's Spark configuration to point to these JARs. This allows you to call into your Scala Spark driver code from the PySpark Python code. Here is an example of how you can configure a Flyte Spark task to run a Scala Spark job:

from flytekit import task
from flytekitplugins.spark import Spark

@task(
    task_config=Spark(
        spark_conf={
            "spark.driver.extraClassPath": "/path/to/your/jarfile.jar",
            "spark.executor.extraClassPath": "/path/to/your/jarfile.jar",
        }
    ),
)
def spark_task() -> float:
    sess = flytekit.current_context().spark_session
    return sess.sparkContext._jvm.com.my.scala.package.ScalaDriver.go()

Would you like more details on this setup? Sources: - https://discuss.flyte.org/t/8022179/we-are-trying-to-execute-sparkjobs-written-in-scala-one-stra/ - https://github.com/flyteorg/flyte/pull/718 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.