Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Re-raising this question as I'm not sure why it was seemingly skipped over. Does anyone have any guidance they can provide on this?
<https://flyte-org.slack.com/archives/CP2HDHKE1/p1694026733367549>

I do not follow, are you saying AWS Batch should send workloads to EKS?

This is not supported by AWS Batch plugin in flyte today

Why not just use K8s Array, we know that scales really well

this is like going from k8s -&gt; flyte -&gt; aws back to -&gt; k8s

You can write a flyte agent if this is a different api, but today out of the box this is not supported

We would imagine that would be easily achievable following <https://www.youtube.com/watch?v=n_rRb8u1GSM>

do you know of any integration between Flyte and Batch on top of EKS?

no, this is my team that showed this. this makes it possible to run Flyte jobs on batch - but the current integration used AWS Batch - Submit job interface

this is the aws batch integration - <https://github.com/flyteorg/flyteplugins/blob/c978638d6ffe10344eeed308f882a2b6f7ec490b/go/tasks/plugins/array/awsbatch/executor.go#L2>

here is the python plugin - <https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-batch/tests/test_aws_batch.py>

Outstanding, we will check this. Thanks, <@UNZB4NW3S> :raised_hands:

By the way the vide you folks talked about - says it used AWS Batch on ECS

<@U05PDQK7B0B> / <@U04DSFUUQQN> has there been any further discussion here? Just want to make sure we close anything open-ended here.

I know we had a brief discussion with the AWS folks on this functionality (batch on EKS) almost a year ago now at a conference. The general consensus was that they would be interested in contributed and supporting the work, but follow-ups grew relatively stale.

I would be happy to shepherd this work. I think the auto-scaling components are very cool. And as Ketan mentioned, this is all stuff that is moving AWSBatch closer to what our current maptask plugin already does with a few caveats.

Hey Dan! Great to hear from you! I think we got the information we were seeking from Ketan and are reviewing internally and figuring out our path forward. We'll likely have a clearer answer on that front in a day or two. But thanks so much for checking in.