Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

I’m early in my Flyte Journey, but I want to build a workflow with different containers (easy!) with one team per container.. However one of these steps require much more resources than the rest (something like 32vcpu, 128 GB RAM) and can’t be broken down. Ideally I don’t want a cluster that size enabled all of the time. I have the option to choose pretty much any cloud, would you recommend something like AWS Batch for resources? Or can you configure your K8s cluster enough through Flyte to auto scale this?

while it does support aws batch, i think it’s probably not necessary in this case.  i assume you’re on eks if you’re using aws?

you should be able to run a separate node group that scales down to 0

you will have to deal with that asg scale up time of course

Welcome <@U04HX87HN1L> thank you for sharing your usecase.
this is absolutely possible,
Option 1: Create a K8s cluster with a K8s auto-scaler or even fargate. and Flyte will automatically back-off etc as you scale to the desired size.
Option 2: Use AWS Batch

Each task can be a separate container. you can also mix and match AWS Batch with K8s.

To share between teams, I would highly recommend that you look into <https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/reference_task.html#sphx-glr-auto-core-flyte-basics-reference-task-py|Reference Tasks>

you can even use - Launchplans in different projects.

your usecase sounds like a perfect fit for Flyte.

Awesome, thank. you both <@UNR3C6Y4T> and <@UNZB4NW3S> for your help