In case anyone s encountered this particular management task

Question

In case anyone s encountered this particular management task we d like to switch our S3 bucket that we use for Flyte to one in a new region and it s a fairly straightforward config change I wanted to clarify will redeploying the Flyte backend services after modifying the storage bucket break in progress runs I was hoping that they could continue to execute given the IAM roles will still permit reading from the old bucket but wanted to know if there are any possibilities of breakages if there s a currently executing dynamic workflow for ex

Samhita Alla · Answer

< Ketan kumare3 > would you mind taking a look at this question

Rahul Mehta · Answer

It looks like this is a somewhat dangerous operation from the channel s history but we don t really care if links for intermediate artifacts are broken we use a separate bucket for scratch vs final outputs Mostly curious if this will lead to workflow execution interruptions

Rahul Mehta · Answer

< Ketan kumare3 > < Eduardo Apolinario eapolinario > do you guys have any idea here

Ketan (kumare3) · Answer

ohh sorry just saw this

Rahul Mehta · Answer

Np totally know you guys are swamped slightly smiling face

Ketan (kumare3) · Answer

< Rahul Mehta> what are you changing are you changing the metadata bucket

Rahul Mehta · Answer

Yes

Rahul Mehta · Answer

We want to move it from usw1 gt use1

Ketan (kumare3) · Answer

uhho then they will break everything

Ketan (kumare3) · Answer

if you change data bucket raw data prefix then it iwll continue running

Ketan (kumare3) · Answer

if you change metadata bucket then previous data wont be found

Ketan (kumare3) · Answer

here is what i would do is drain propeller and then transition

Ketan (kumare3) · Answer

that means do not pick up new work but complete all the work

Rahul Mehta · Answer

Ok how would we drain propeller

Rahul Mehta · Answer

To be fair we re treating Flyte as stateless from the POV of other systems any important artifacts are saved in a separate bucket and we also track run state separately we just dont want to impact in progress runs

Ketan (kumare3) · Answer

stop sending crds to the k8s cluster

Ketan (kumare3) · Answer

just create a new cluster that uses new metadata bucket and send workloads there

Ketan (kumare3) · Answer

ya so if you do 2 cluster migration then old cluster can be drained and deleted

Ketan (kumare3) · Answer

that should not be impacted

Rahul Mehta · Answer

I see got it Even if we sync over the s3 bucket it ll break

Rahul Mehta · Answer

we could take a small downtime window to sync things over and redeploy

Ketan (kumare3) · Answer

no it will not

Ketan (kumare3) · Answer

if you can sync one option then would be to make global s3 buckets possible

Ketan (kumare3) · Answer

but it relies on previous nodes data to be available

Ketan (kumare3) · Answer

it uses tricks like new keys to also maintain consistency but s3 is now fully consistent

Rahul Mehta · Answer

ok but to confirm changing the bucket name won t impact it I wanted to confirm the state in the db just stores keys and not the full `s3 lt bucket name gt ` path

Rahul Mehta · Answer

since then the sync strategy wouldn t work

Derek Yu · Answer

Hi Also following this thread Similarly been running into issues trying to change the metadata bucket gcs Discussion <https flyte org slack com archives CP2HDHKE1 p1684301052938649|here> After creating a new bucket copying all data over from the old bucket and using the new bucket all historical workflows are showing Status code 500 Failed to fetch data full error message in thread Is it not possible to change buckets and be able to access historical workflow data in the console

Ketan (kumare3) · Answer

I do not think so as the bucket name has changed The db stores links to the bucket You will have to run a migration on the db

Derek Yu · Answer

I see thanks Ketan And seems like even keeping the old bucket up flyte still can t access it via the old links Was thinking it might work by just keeping both buckets up

Ketan (kumare3) · Answer

it should

Ketan (kumare3) · Answer

you have to enable multi container mode in the storage config

Derek Yu · Answer

it worked thanks for your help Ketan

Rahul Mehta · Answer

What config changes did you need to make < Derek Yu> We re looking to do the exact same thing

Derek Yu · Answer

Add this block to your helm values <https docs flyte org en latest deployment configuration generated datacatalog config html enable multicontainer bool|this> setting ``` storage enableMultiContainer true``` And update flyte to use your new bucket I believe it will continue reading old workflows from the old bucket while using the new bucket for new workflows

Rahul Mehta · Answer

Awesome Thank you slightly smiling face

Rahul Mehta · Answer

Going to give this a try today