Channels
#ask-the-community
Title
# ask-the-community
m
Matt Coan
09/08/2023, 6:36 PMRe-raising this question as I'm not sure why it was seemingly skipped over. Does anyone have any guidance they can provide on this?
https://flyte-org.slack.com/archives/CP2HDHKE1/p1694026733367549
k
Ketan (kumare3)
09/08/2023, 7:15 PM
I do not follow, are you saying AWS Batch should send workloads to EKS?
This is not supported by AWS Batch plugin in flyte today
Why not just use K8s Array, we know that scales really well
this is like going from k8s -> flyte -> aws back to -> k8s
You can write a flyte agent if this is a different api, but today out of the box this is not supported
do you know of any integration between Flyte and Batch on top of EKS?
k
Ketan (kumare3)
09/08/2023, 8:47 PM
no, this is my team that showed this. this makes it possible to run Flyte jobs on batch - but the current integration used AWS Batch - Submit job interface
this is the aws batch integration - https://github.com/flyteorg/flyteplugins/blob/c978638d6ffe10344eeed308f882a2b6f7ec490b/go/tasks/plugins/array/awsbatch/executor.go#L2
here is the python plugin - https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-batch/tests/test_aws_batch.py
j
Juan Irabedra
09/08/2023, 9:48 PMOutstanding, we will check this. Thanks, @Ketan (kumare3) 🙌
k
Ketan (kumare3)
09/08/2023, 10:45 PM
By the way the vide you folks talked about - says it used AWS Batch on ECS
from EKS (where flyte runs)
d
Dan Rammer (hamersaw)
09/11/2023, 4:26 PM@Juan Irabedra / @Matt Coan has there been any further discussion here? Just want to make sure we close anything open-ended here.
I know we had a brief discussion with the AWS folks on this functionality (batch on EKS) almost a year ago now at a conference. The general consensus was that they would be interested in contributed and supporting the work, but follow-ups grew relatively stale.
I would be happy to shepherd this work. I think the auto-scaling components are very cool. And as Ketan mentioned, this is all stuff that is moving AWSBatch closer to what our current maptask plugin already does with a few caveats.
m
Matt Coan
09/11/2023, 4:33 PMHey Dan! Great to hear from you! I think we got the information we were seeking from Ketan and are reviewing internally and figuring out our path forward. We'll likely have a clearer answer on that front in a day or two. But thanks so much for checking in.
2 Views