Hi, is it possible to change the bucket that flyte uses? If so, are there migration instructions? We tried, and on the console, see 500 errors reading data from the old bucket for old workflows

Have you tried configuring raw output data prefix? If you're using

pyflyte register

, here's the CLI reference: https://docs.flyte.org/projects/flytekit/en/latest/pyflyte.html#pyflyte-register.

Thanks, hm no haven't yet, we use

flytctl register

I believe. Can you elaborate on how the raw output data prefix works and how that might help with migrating flyte to use a new bucket?

https://docs.flyte.org/en/latest/concepts/data_management.html#raw-data-prefix -- should help you understand what raw data prefix is all about.

Hi Samhita, it seems like raw data output is for writing data to a bucket. The problem we are facing is when we changed our bucket (and bucket name) that flyte uses for storage, the old workflows (pre-migration) are attempting to grab the data from the new bucket to display on the console. Even though all the data was copied over to the new bucket, it somehow cannot get the data from the new bucket and displays a "404 Not found" on flyte console. Thus, we cannot see any historical workflow runs. How would raw output data prefix help here? Do you know of anyone trying to change bucket names? We are wondering if it's even possible in the first place, and still be able to display historical workflows.

Changing the bucket name should be possible. Are you encountering issues when you're trying to run old workflows that have already been registered pre-migration?

Hi @Samhita Alla! We’re encountering issues when trying to pull up old executions in console. The list of executions is fine, but once you click on one it won’t work (because some bucket name comparison fails).

cc @Yee

Working on that 👍 We had to roll back our production instance, trying to reproduce on our staging Flyte deployment now

Hi @Samhita Alla sorry for the delay. We circled around and were able to replicate this. Steps taken: 1. Transferred all data from old bucket to new bucket 2. Updated the bucket name in the helm values here with the new bucket Now, all historical workflows are showing "*Failed to fetch data*". The exact error:

Request failed with status code 500 Unable to read WorkflowClosure from location <gs://old_bucket/metadata/admin/project/development/flyte.project.x.foo_workflow/asdf> : path:<gs://old_bucket/metadata/admin/project/development/flyte.project.x.foo_workflow/asdf>: Conf container:new_bucket != Passed Container:old_bucket. Dynamic loading is disabled: not found

It looks like it's trying to access data from the old bucket still. (The old bucket still exists). What's interesting is that last line:

Conf container:new_bucket != Passed Container:old_bucket. Dynamic loading is disabled: not found

Thanks for your help!

I believe changing metadata bucket isn't a straightforward process. Please take a look at this thread: https://flyte-org.slack.com/archives/C01P3B761A6/p1686933880822739?thread_ts=1686237668.672569&cid=C01P3B761A6

thanks, I've followed up there as well!