New RunLLM response generated:
>>>*# Question*
Hey team! In our current Flyte -> Spark setup, we use our internal infra to run Spark jobs. This infra is owned by another team.
# Answer
It seems likely that you are looking to integrate Flyte with an existing Spark infrastructure managed by another team. My understanding is that Flyte can execute Spark jobs natively on a Kubernetes cluster, leveraging the Spark On K8s Operator, which allows for managing a virtual cluster's lifecycle. However, if you want to submit Spark jobs to a different Kubernetes cluster, it might require custom configurations or plugins.
Would you like more details on how to potentially configure Flyte to submit Spark jobs to a different Kubernetes cluster?
Sources:
-
Flyte and Spark
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732037131.061239 👍 0 button 👎 0 button 💡
Hint: Mention @runllm in the thread for followups.