https://flyte.org logo
Join Slack
Powered by
In case anyone's encountered this particular manag...
# flyte-deployment
r
In case anyone's encountered this particular management task: we'd like to switch our S3 bucket that we use for Flyte to one in a new region, and it's a fairly straightforward config change - I wanted to clarify, will redeploying the Flyte backend services after modifying the storage bucket break in-progress runs? I was hoping that they could continue to execute given the IAM roles will still permit reading from the old bucket, but wanted to know if there are any possibilities of breakages if there's a currently-executing dynamic workflow (for ex)
s
@Ketan (kumare3), would you mind taking a look at this question?
r
It looks like this is a somewhat dangerous operation from the channel’s history, but we don’t really care if links for intermediate artifacts are broken (we use a separate bucket for scratch vs final outputs). Mostly curious if this will lead to workflow execution interruptions.
@Ketan (kumare3) / @Eduardo Apolinario (eapolinario) do you guys have any idea here?
k
ohh sorry just saw this
r
Np, totally know you guys are swamped 🙂
k
@Rahul Mehta what are you changing, are you changing the metadata bucket?
r
Yes
We want to move it from usw1 -> use1
k
uhho, then they will break everything
if you change data bucket (raw data prefix) - then it iwll continue running
if you change metadata bucket, then previous data wont be found
here is what i would do, is drain propeller and then transition
that means, do not pick up new work, but complete all the work
r
Ok - how would we "drain" propeller?
To be fair, we're treating Flyte as "stateless" from the POV of other systems, any important artifacts are saved in a separate bucket and we also track run state separately - we just dont want to impact in-progress runs
k
stop sending crds to the k8s cluster
just create a new cluster, that uses new metadata bucket and send workloads there
ya so if you do 2 cluster migration, then old cluster can be drained and deleted
that should not be impacted
r
I see - got it. Even if we sync over the s3 bucket it'll break?
we could take a small downtime window to sync things over and redeploy
k
no it will not
if you can sync, one option then would be to make global s3 buckets possible
but, it relies on previous nodes data to be available
it uses tricks like new keys to also maintain consistency, but s3 is now fully consistent
r
ok. but to confirm, changing the bucket name won't impact it. I wanted to confirm the state in the db just stores keys and not the full 
s3://<bucket name>/...
path
since then the sync strategy wouldn't work
d
Hi! Also following this thread. Similarly been running into issues trying to change the metadata bucket (gcs). Discussion here. After creating a new bucket, copying all data over from the old bucket, and using the new bucket, all historical workflows are showing "Status code 500: Failed to fetch data". (full error message in thread) Is it not possible to change buckets and be able to access historical workflow data in the console?
k
I do not think so, as the bucket name has changed. The db stores links to the bucket. You will have to run a migration on the db
d
I see thanks Ketan. And seems like even keeping the old bucket up, flyte still can't access it via the old links Was thinking it might work by just keeping both buckets up
k
it should
you have to enable multi-container mode in the storage config
d
it worked! thanks for your help Ketan!
r
What config changes did you need to make @Derek Yu? We're looking to do the exact same thing
d
Add this block to your helm values (this setting):
Copy code
storage:
    enableMultiContainer: true
And update flyte to use your new bucket. I believe it will continue reading old workflows from the old bucket while using the new bucket for new workflows
r
Awesome! Thank you 🙂
Going to give this a try today
149 Views
Open in Slack
PreviousNext
Powered by