Has anybody used the Spark plugin with volcano or another batch scheduler? I see some docs on volcano with kubeflow but nothing that also interacts with a plugin
c
clean-glass-36808
08/26/2024, 9:35 PM
Not sure if its helpful but I've implemented a custom plugin which schedules Python tasks to the Armada batch scheduler.
f
freezing-airport-6809
08/27/2024, 3:56 AM
@clean-glass-36808 this is different because Joshua wants to use gang scheduling with spark. We have it available in union