Hey folks. I was wondering if Flyte has support for writing spark jobs in Scala. I see a lot of examples for pyspark using the python flyte api, but I want to use flyte-scala to write scala spark pipelines, and documentation seems kinda sparse on this topic.

this does not exist yet

yes I do!

ya the first 2 should be trivial, but databricks and dataproc plugins need to be implemented

I took a quick look. I think the Databricks one might be harder as they don’t have a java api that I can find. Dataproc of course does have one though, and I’m guessing the implementation would be somewhat similar to the Dataflow one we already have. Would need guidance on K8. cc@important-train-84785

K8 should not be hard

So the folks you mentioned above are probably the actual owners of the Flyte infrastructure within Spotify. I am working with Mark (mentioned above) who is closer to them organizationally than I am. I am just a motivated potential user. But given that Spark is not a “blessed” technology within Spotify yet, unsure as to their roadmap. I’d be happy to take a look into this myself if I had guidance from one of the above though. Might require an internal discussion as I don’t know the code structure well enough yet to know how much of this is a potential open source contribution, and how much is Spotify specific.

More from a point of view to understand what is the goal, how to maximize impact in short term

Will get back in the next week or two? Have to have some internal discussions first, but I still really wanna do this.

this is fantastic