I have deployed the Flyte control plane and data plane components in two different clusters, and they are using two different Minio Buckets. I noticed that I cannot visualize tasks' input and outputs or recover Workflow runs even though the control plane components have read-only access to the data plane Bucket. This is the error log I found in FlyteAdmin.

{"json":{"exec_id":"atsr9qglqwgmsfjc4fzx","node":"n1"},"level":"error","msg":"Failed to read from the raw store [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-atsr9qglqwgmsfjc4fzx/n1/data/inputs.pb>] Error: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:34:35Z"}
{"json":{"exec_id":"awg84z5mvwbmql2qrc9n","node":"n0"},"level":"warning","msg":"Failed to read inputs from URI [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/inputs.pb>] with err: path:<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/inputs.pb>: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:46:10Z"}
{"json":{"exec_id":"awg84z5mvwbmql2qrc9n","node":"n0"},"level":"error","msg":"Failed to read from the raw store [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/0/outputs.pb>] Error: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:46:10Z"}

If the control plane uses a different bucket from the Propeller, it refuses to load anything. Can we circumvent this behaviour? I was not expecting the "Recover" feature to also stop working; if I understand correctly, the control plane is trying to "bind" the outputs of the previously successful workflow nodes while recovering, and it is not proceeding because of the mismatching Bucket name.