Hi Flyters, A n00b question: We have written 10+ Spark applications written in Scala and currently deploying them with some other orchastration tool. We want to use Flyte for orchastration purpose. Some people told us that to use Flyte we have to REWRITE all these 10+ applications using PySpark. Is it really true that Flyte does not support Spark applications written in Scala?

hey @Chandrashekhar Kotekar it is not that Flyte does not support spark in scala, it is just that the native flytekit implementation is available in python today here is the python SDK - for writing pyspark - https://github.com/flyteorg/flytekit/blob/28da983bba36e243bc671f9ba1aa53a0791efd62/plugins/flytekit-spark/flytekitplugins/spark/task.py#L118 Infact it is possible to use python sdk to run scala spark jobs - that work has not been done (please contribute) - here is a strawman PR for the same - https://github.com/flyteorg/flytekit/pull/767/files Another option is to use the Java / Scala SDK https://github.com/flyteorg/flytekit-java/tree/master It does not have the JAVA spark plugin written. (again can be contributed) Happy to be a consultant in some of these

@Ketan (kumare3) Thank you for quick answers 🙂 (for some reason I am not able to reply your messages directly hence summing up my answers here) • If PySpark SDK can run jars (Scala + Spark jobs) then our problem is mostly solved - we don't have to rewrite all those applications again. I will have a look at the PR • and yes we want to run Spark on K8s

@Chandrashekhar Kotekar contributions welcome ❤️