Re-raising this question as I'm not sure why it wa...
# ask-the-communitym
Matt Coan
09/08/2023, 6:36 PMRe-raising this question as I'm not sure why it was seemingly skipped over. Does anyone have any guidance they can provide on this?
https://flyte-org.slack.com/archives/CP2HDHKE1/p1694026733367549
k
Ketan (kumare3)
09/08/2023, 7:15 PM
I do not follow, are you saying AWS Batch should send workloads to EKS?
Ketan (kumare3)
09/08/2023, 7:15 PM
This is not supported by AWS Batch plugin in flyte today
Ketan (kumare3)
09/08/2023, 7:15 PM
Why not just use K8s Array, we know that scales really well
Ketan (kumare3)
09/08/2023, 7:15 PM
this is like going from k8s -> flyte -> aws back to -> k8s
Ketan (kumare3)
09/08/2023, 7:16 PM
You can write a flyte agent if this is a different api, but today out of the box this is not supported
Juan Irabedra
09/08/2023, 7:25 PMdo you know of any integration between Flyte and Batch on top of EKS?
k
Ketan (kumare3)
09/08/2023, 8:47 PM
no, this is my team that showed this. this makes it possible to run Flyte jobs on batch - but the current integration used AWS Batch - Submit job interface
Ketan (kumare3)
09/08/2023, 8:49 PM
this is the aws batch integration - https://github.com/flyteorg/flyteplugins/blob/c978638d6ffe10344eeed308f882a2b6f7ec490b/go/tasks/plugins/array/awsbatch/executor.go#L2
Ketan (kumare3)
09/08/2023, 8:49 PM
here is the python plugin - https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-batch/tests/test_aws_batch.py
j
Juan Irabedra
09/08/2023, 9:48 PMOutstanding, we will check this. Thanks, @Ketan (kumare3) 🙌
k
Ketan (kumare3)
09/08/2023, 10:45 PM
By the way the vide you folks talked about - says it used AWS Batch on ECS
Ketan (kumare3)
09/08/2023, 10:45 PM
from EKS (where flyte runs)
d
Dan Rammer (hamersaw)
09/11/2023, 4:26 PM@Juan Irabedra / @Matt Coan has there been any further discussion here? Just want to make sure we close anything open-ended here.
I know we had a brief discussion with the AWS folks on this functionality (batch on EKS) almost a year ago now at a conference. The general consensus was that they would be interested in contributed and supporting the work, but follow-ups grew relatively stale.
I would be happy to shepherd this work. I think the auto-scaling components are very cool. And as Ketan mentioned, this is all stuff that is moving AWSBatch closer to what our current maptask plugin already does with a few caveats.
m
Matt Coan
09/11/2023, 4:33 PMHey Dan! Great to hear from you! I think we got the information we were seeking from Ketan and are reviewing internally and figuring out our path forward. We'll likely have a clearer answer on that front in a day or two. But thanks so much for checking in.
2 Views