Hi, is it possible to change the bucket that flyte...
# ask-the-community
d
Hi, is it possible to change the bucket that flyte uses? If so, are there migration instructions? We tried, and on the console, see 500 errors reading data from the old bucket for old workflows
s
Have you tried configuring raw output data prefix? If you're using
pyflyte register
, here's the CLI reference: https://docs.flyte.org/projects/flytekit/en/latest/pyflyte.html#pyflyte-register.
d
Thanks, hm no haven't yet, we use
flytctl register
I believe. Can you elaborate on how the raw output data prefix works and how that might help with migrating flyte to use a new bucket?
s
https://docs.flyte.org/en/latest/concepts/data_management.html#raw-data-prefix -- should help you understand what raw data prefix is all about.
d
Hi Samhita, it seems like raw data output is for writing data to a bucket. The problem we are facing is when we changed our bucket (and bucket name) that flyte uses for storage, the old workflows (pre-migration) are attempting to grab the data from the new bucket to display on the console. Even though all the data was copied over to the new bucket, it somehow cannot get the data from the new bucket and displays a "404 Not found" on flyte console. Thus, we cannot see any historical workflow runs. How would raw output data prefix help here? Do you know of anyone trying to change bucket names? We are wondering if it's even possible in the first place, and still be able to display historical workflows.
s
Changing the bucket name should be possible. Are you encountering issues when you're trying to run old workflows that have already been registered pre-migration?
Have you tried registering your workflows now? Also would you mind sharing the error you're seeing?
b
Hi @Samhita Alla! We’re encountering issues when trying to pull up old executions in console. The list of executions is fine, but once you click on one it won’t work (because some bucket name comparison fails).
s
cc @Yee
Can you share with us the exact error you're seeing?
b
Working on that 👍 We had to roll back our production instance, trying to reproduce on our staging Flyte deployment now
d
Hi @Samhita Alla sorry for the delay. We circled around and were able to replicate this. Steps taken: 1. Transferred all data from old bucket to new bucket 2. Updated the bucket name in the helm values here with the new bucket Now, all historical workflows are showing "*Failed to fetch data*". The exact error:
Copy code
Request failed with status code 500 Unable to read WorkflowClosure from location <gs://old_bucket/metadata/admin/project/development/flyte.project.x.foo_workflow/asdf> : path:<gs://old_bucket/metadata/admin/project/development/flyte.project.x.foo_workflow/asdf>: Conf container:new_bucket != Passed Container:old_bucket. Dynamic loading is disabled: not found
It looks like it's trying to access data from the old bucket still. (The old bucket still exists). What's interesting is that last line:
Copy code
Conf container:new_bucket != Passed Container:old_bucket. Dynamic loading is disabled: not found
Thanks for your help!
Wondering still perhaps if there may be some other references to the bucket that need to be updated..
s
I believe changing metadata bucket isn't a straightforward process. Please take a look at this thread: https://flyte-org.slack.com/archives/C01P3B761A6/p1686933880822739?thread_ts=1686237668.672569&amp;cid=C01P3B761A6
d
thanks, I've followed up there as well!
hey @Samhita Alla, it worked after we turned on multiContainer mode. Thanks for your help again!
105 Views