I was looking through the flyte config structs, and flytestdlib uses a StowConfig. Looking at the stow repo it doesn't look like that project has released in over a year and there are no PRs or issues open. Is stow still a supported part of flyte?

yes it is, we are aware that it’s been stale for a while but have not had the resources to investigate an alternative. fortunately though we’ve not had any issues. is this something you just happened to notice or did you run across this because of a specific issue?

We are very early in our flyte investigation. We've installed enterprise deployment recommended in the installation docs, but it's a singular pod which looks to smash together all the other flyte operators. We've been looking at how to stand up the bits we need to eval, like storage, so digging through various config structs to figure things out. That's where we ran across StowConfig and when to look at that service and how to configure it.

got it

How many jobs can 1 controller handle? I know it spawns kube pods for the work. Our scale could be thousands a day. We hope to eventually do some scale testing but need to get it all stood up first. 🙂

The original stow repo has been deprecated but we've forked Stow as you could tell in github.com/flyteorg/stow and are maintaining it for our needs ..

Oh, that flyteorg/stow was the repo I was looking at. I didn't realize it had been forked from something else

RE: Scale, think 10s of thousands... without naming names, I know companies running 20-30k executions on a single controller... The controller (propeller) can also run in a sharded mode (we call it propeller-manager) that can scale virtually infinitely. (Your k8s cluster will start having trouble with etcd. or api server scale long before it does)

I saw the propeller controller references in the codebase. I don't yet understand how a propeller instance gets spawned or under what conditions.