Is there a way to configure the storage location prefix for Flyte #announcements

Is there a way to configure the storage location/p...

Sören Brunk

05/09/2022, 12:35 PM

Is there a way to configure the storage location/prefix for code when using the new fast registration without bucket credentials? It used to be stored under

bucket-name/fast/

before (not sure if I had configured it before, I can't find any config related to that). Now it's just stored in random paths under the bucket root.

Ketan (kumare3)

05/09/2022, 2:31 PM

You can configure the prefix - https://github.com/flyteorg/flyteadmin/blob/072ce740118be58b57e8e6e26ea552368f31ac9e/pkg/config/config.go#L39

🙏 1

Ketan (kumare3)

05/09/2022, 2:40 PM

But the storage location is where Flyte stores all it's metadata

Sören Brunk

05/09/2022, 3:00 PM

That's fine, I just want to have it under a prefix, not the bucket root

Sören Brunk

05/25/2022, 2:28 PM

Hmm perhaps I misunderstood how this works. If I set

storagePrefix: fast

, flyte still creates random directories at the root of the bucket, and THEN prepends the prefix like:

<gs://bucket/jl/fast/project/domain/><random>/<random>.tar.gz

Sören Brunk

05/25/2022, 2:31 PM

Using an absolute prefix

storagePrefix: "<gs://bucket/fast>"

doesn't work either.

Ketan (kumare3)

05/25/2022, 2:31 PM

Ya we can stop doing the random can you file an issue

Ketan (kumare3)

05/25/2022, 2:31 PM

It's using the same sharded

Sören Brunk

05/25/2022, 2:33 PM

Ok I'll create an issue

Sören Brunk

05/25/2022, 7:46 PM

I realized my issue is just about the prefix order, not sure about the random so I left that out for now. Here’s the issue: https://github.com/flyteorg/flyte/issues/2545

169 Views

Open in Slack

Previous Next