I’m early in my Flyte Journey, but I want to build a workflow with different containers (easy!) with one team per container.. However one of these steps require much more resources than the rest (something like 32vcpu, 128 GB RAM) and can’t be broken down. Ideally I don’t want a cluster that size enabled all of the time. I have the option to choose pretty much any cloud, would you recommend something like AWS Batch for resources? Or can you configure your K8s cluster enough through Flyte to auto scale this?

hey @Niklas Bivald

Welcome @Niklas Bivald thank you for sharing your usecase. this is absolutely possible, Option 1: Create a K8s cluster with a K8s auto-scaler or even fargate. and Flyte will automatically back-off etc as you scale to the desired size. Option 2: Use AWS Batch Each task can be a separate container. you can also mix and match AWS Batch with K8s. To share between teams, I would highly recommend that you look into Reference Tasks you can even use - Launchplans in different projects. your usecase sounds like a perfect fit for Flyte.

Awesome, thank. you both @Yee and @Ketan (kumare3) for your help