Cluster B (Propeller) needs access to S3 from cluster A because Flyte stores workflow metadata (including references to raw data) in the control plane's S3 bucket. When Propeller in cluster B executes tasks, it reads input/output data paths from metadata written by Admin (cluster A), which point to S3 bucket A. This means Propeller must access S3 A to fetch or store workflow data, even if it has its own S3 configured locally. There is no documented way to avoid this cross-bucket access, as Flyte's architecture assumes a shared object store for workflow data across clusters. Using separate S3 buckets for each cluster without cross-access will result in errors or missing data during execution and UI inspection, as seen in this issue. To avoid this, you must either (1) ensure all clusters use the same S3 bucket for Flyte data, or (2) manually update all metadata references in the Flyte database to point to the new bucket after migration, which is not automated and can be error-prone. There is no built-in support for fully isolated S3 buckets per cluster in a multicluster Flyte deployment according to the available documentation and issues. Would you like more detail on possible migration or workaround strategies? Sources: - FlyteAdmin Continues Reading from Old S3 Bucket After Migration, Causing 500 Errors in UI 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

It seems likely that most workflow data (inputs, outputs, artifacts) is stored in the S3 bucket configured for the control plane, not the data plane. Flyte stores metadata (including references to raw data) in the control plane's S3, and task input/output data paths are written there. The code executed (workflow definitions, task specs) and metadata are also stored in the control plane's S3. When a task runs in the data plane, it reads/writes data to the S3 bucket referenced in the metadata, which is typically the control plane's S3, unless you have customized the output prefix per execution (which is not the default and requires careful configuration). The data plane's S3 is not used unless specifically configured for raw data output, but this is not standard and can cause issues with data accessibility and UI errors if not coordinated. Specifically: - Code executed (workflow/task definitions): stored in control plane S3. - Input/output data (workflow and tasks): by default, stored in control plane S3, referenced by metadata. - Metadata (references, small values): stored in control plane S3. - Data plane S3 is only used if you explicitly set output prefixes to point there, but this is not the default and can break cross-cluster access and UI features if not handled carefully. Would you like more detail on how to configure per-execution output prefixes or on Flyte's data flow? Sources: - https://flyte.org/blog/speed-up-time-to-production-with-flyte-on-eks-from-automation-to-multicloud#starCount 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 how to configure data plane to use its own bucket then? And AFAIU even if I configure it I still need to give propeller access to S3 on control plane, correct?

To configure the data plane (Propeller) to use its own S3 bucket, set the

rawoutput-prefix

in the Propeller configuration to point to the desired bucket (e.g.,

rawoutput-prefix: s3://<data-plane-bucket>/

). This can be done in the Helm chart or ConfigMap for Propeller. However, even with this configuration, Propeller still needs access to the control plane's S3 bucket because workflow metadata, code, and some input references are stored there by FlyteAdmin. Propeller must read these to execute workflows and tasks, as confirmed in this issue. So yes, you must grant Propeller access to both its own S3 bucket (for outputs) and the control plane's S3 bucket (for metadata and workflow artifacts). There is no supported way to fully isolate S3 access per cluster without breaking Flyte's data flow and UI features. Sources: - FlyteAdmin Continues Reading from Old S3 Bucket After Migration, Causing 500 Errors in UI - Flyte Helm chart values: rawoutput-prefix 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

Yes, that's correct. In the official Helm charts,

rawoutput-prefix

defaults to

s3://{{ .Values.storage.bucketName }}/

, so if you set

storage.bucketName

, both metadata and user data (raw outputs) will be stored in that bucket by default unless you override

rawoutput-prefix

separately. This means Propeller will use the bucket defined in

storage.bucketName

for user data storage unless explicitly configured otherwise. See flyte-core values.yaml and PR #6433. Would you like more detail on customizing these settings? Sources: - flyte-core values.yaml - flyte-core: Default propeller `rawoutput-prefix` to use `storage.bucketName` 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

The IAM role assumed by the pod is the one configured in the data plane (Propeller) cluster, not the control plane (Admin) cluster. When a workflow task is executed, the pod is created in the data plane's Kubernetes cluster, and it uses the

defaultIamRole

annotation from the data plane's namespace/service account configuration. The control plane's IAM role is not used for pod execution. This is confirmed by the Flyte deployment and configuration logic, which applies the IAM role annotation to the service account in the data plane where the pod actually runs (source). Would you like more detail on how IAM roles are mapped to service accounts in Flyte? Sources: - Flyte on EKS: IAM Roles for Service Accounts 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

You are correct that, in Flyte's default multi-cluster setup, the

defaultIamRole

and other resource templates are typically managed and distributed from the control plane (Admin) to the data plane (Propeller) clusters. The control plane's configuration (including templated values like

defaultIamRole

) is used to create or update the Kubernetes resources (such as service accounts with IAM role annotations) in the data plane via the cluster resource sync mechanism. As a result, the worker pod in the data plane may end up using the IAM role defined in the control plane's configuration, not a value set independently in the data plane. This behavior is by design: the control plane's

cluster_resources.yaml

(or equivalent Helm values) is the source of truth for project/domain resource quotas and IAM roles, and these are propagated to all data planes. If you want different IAM roles per data plane, you must customize the control plane's configuration to specify different roles for each cluster, which is not supported out-of-the-box and would require manual intervention or custom automation (source). Would you like more detail on how to customize IAM roles per data plane or how the cluster resource sync works? Sources: - flyte-core values-eks.yaml: cluster_resources and defaultIamRole 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

It seems likely that the worker pod (in the data plane) primarily needs access to the S3 bucket managed by the control plane (Admin), since workflow metadata and raw data paths are written there and referenced during execution. Even if the IAM role is defined by the control plane, the pod must access the S3 bucket where FlyteAdmin stores workflow data. The S3 bucket managed by Propeller (data plane) is only used if you explicitly configure output prefixes to point there, which is not the default. So, by default, the worker pod needs access to the S3 bucket managed by Admin, not the one managed by Propeller, unless you have customized the storage configuration. Would you like more detail on how to configure output prefixes for data plane buckets? Sources: - https://flyte.org/blog/speed-up-time-to-production-with-flyte-on-eks-from-automation-to-multicloud#starCount - https://github.com/flyteorg/flyte/pull/3994 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.