A n00b question: We have written 10+ Spark applications written in Scala and currently deploying them with some other orchastration tool. We want to use Flyte for orchastration purpose.
Some people told us that to use Flyte we have to REWRITE all these 10+ applications using PySpark.
Is it really true that Flyte does not support Spark applications written in Scala?
@Ketan (kumare3) Thank you for quick answers 🙂 (for some reason I am not able to reply your messages directly hence summing up my answers here)
• If PySpark SDK can run jars (Scala + Spark jobs) then our problem is mostly solved - we don't have to rewrite all those applications again. I will have a look at the PR
• and yes we want to run Spark on K8s