hi all we currently run both our Flyte clusters in AWS EKS d Flyte #flyte-deployment

hi all - we currently run both our Flyte clusters ...

Brian Tang

11/08/2023, 8:00 AM

hi all - we currently run both our Flyte clusters in AWS EKS (dev, prod). one of our clients wants us to run their ML workloads in Huawei cloud. looking at the helm charts, it looks like only

s3

and

gcs

are the only accepted

provider

for object storage. any advice on how we can deploy Flyte in Huawei cloud? has anyone done it before? my understanding is, flyte-binary requires postgres db and 2 object storage buckets. almost everything else is kubernetes-native?

Chris Grass

11/08/2023, 2:35 PM

Hi Brian - i created a PR recently that formalizes azure storage in

flyte-binary

. i think you can also use the

custom

type if config is just a passthrough to storage via stow, but that lib might need an implementation for the huawei client.

Chris Grass

11/08/2023, 2:43 PM

your issue brings up a good point for the community to discuss though - does the configuration in flyte-binary need to be implementation specific, or should it be a generic passthrough for stow? i started a conversation here about whether or not the product-specific storage clients should be used directly in flyte or if that work should be offloaded to stow. flyte might feel a little bloated if there was an implementation for every cloud storage provider (which would also likely be duplicated in stow)

Ketan (kumare3)

11/08/2023, 3:24 PM

@Brian Tang that is right- but no one has tried Flyte in huawei

Brian Tang

11/09/2023, 2:58 AM

thanks @Chris Grass. so essentially a huawei implementation in stow + having it merged into the forked stow repo + support in the flyte-binary configmap (either having a kind

huawei

or using

custom

) what about deploying minio as a service in the cluster and exposing those “buckets” to flyte. this isn’t ideal, but i suppose that’s an alternative to solving the object storage problem?

Ketan (kumare3)

11/09/2023, 4:42 AM

@Brian Tang that should work out of the box? Does it now. Minio is what is used in sandbox

Chris Grass

11/09/2023, 3:02 PM

yeah, i think those are the two options. the stow huawei implementation is non-trivial but should be possible. minio is certainly the easier option if it's available to you

Ketan (kumare3)

11/09/2023, 3:12 PM

You don’t need a full stow client implementation ideally, if you look at flytestdlib.storage, only get, put, head and signedurl are used

2 Views

Open in Slack

Previous Next