@ancient-wolf-19325 I want to enable

enableDynamicContainerLoading

option of StowStore setting here: https://github.com/flyteorg/flyte/blob/d50fcb8402ca9ac4d2befa5a338b76b765c4d8e2/flytestdlib/storage/stow_store.go#L146 How do I do that thru helmchart? can you give me some documentation around that

The Flyte's documentation I have ingested does not mention how to enable the

enableDynamicContainerLoading

option of StowStore via the Helm chart or provide documentation specifically about configuring this setting through Helm. If you can clarify your deployment context (e.g., which storage backend you are using, or if you are customizing the storage configuration in your values.yaml), it may help in providing more targeted guidance. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I am using S3 as a backend and deploying multi-cluster setup via flyte-core helmchart

There's no Flyte documentation I have ingested mentioning how to enable the

enableDynamicContainerLoading

option of StowStore specifically when using S3 as a backend in a multi-cluster setup via the flyte-core Helm chart. If you can provide more details about where you want this option to take effect (e.g., FlyteAdmin, FlytePropeller, or both), or if you have a custom storage.yaml, that context could help in determining if a workaround or manual configuration is possible. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I ran into this problem that told me to enable it: https://discuss.flyte.org/t/13149329/our-inputs-and-outputs-are-always-empty-why-is-this-the-task

The Flyte documentation I have ingested does not mention how to enable the

enableDynamicContainerLoading

option of StowStore in response to issues where task inputs and outputs are always empty, nor does it provide step-by-step instructions for resolving this specific problem via the Helm chart or configuration files. If you can share the exact error message or configuration snippet you are using, or clarify which Flyte component (e.g., FlytePropeller, FlyteAdmin) is reporting the issue, it may help in diagnosing the root cause or suggesting a workaround. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 in multicluster setup, Ive an admin bucket and bucket for each individual propeller. what are permissions required and why, can yo udraw me out it somehow?

It seems likely that in a Flyte multi-cluster setup with an admin bucket and separate buckets for each FlytePropeller, each Propeller (data plane) needs permissions to read/write only its own bucket, while FlyteAdmin (control plane) needs access to the admin/metadata bucket. This separation allows for least-privilege access and isolation between clusters. Each Propeller's service account should have IAM permissions for its bucket (e.g., s3:GetObject, s3:PutObject, s3:ListBucket), and FlyteAdmin's service account should have similar permissions for the admin bucket. This model is described in the Flyte reference architecture for EKS, which shows IAM roles mapped to service accounts for secure access to S3 buckets, ensuring that each component can only access the resources it needs for workflow execution and metadata management. For a visual, see the reference implementation's diagram and IAM policy example in the linked blog post. Would you like a step-by-step breakdown or a diagram of the permission model? Sources: - https://flyte.org/blog/speed-up-time-to-production-with-flyte-on-eks-from-automation-to-multicloud#starCount - https://github.com/flyteorg/flyte/issues/5051 Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Seems like I am having a problem - since control plane does not have access to dataplane buckets I am getting 403's while trying to shot the inputs and outputs of workflow results in flyt econsole. Is that expected or somehow workaroundable without permissions?

Yes, this is expected: in a multi-cluster Flyte setup, the control plane (FlyteAdmin) does not have access to data plane (Propeller) buckets by design, which leads to 403 errors when FlyteConsole tries to fetch workflow inputs/outputs stored in those buckets. This separation is intentional for security and isolation, as discussed in community threads and GitHub issues. There is currently no built-in workaround to view these artifacts in the console without granting the control plane access to the data plane buckets; Flyte relies on bucket permissions for data access, and FlyteAdmin cannot fetch data it cannot read. For more details, see the discussion and explanation in Flyte GitHub Issue #5051. Would you like more detail on possible architectural workarounds or security implications? Sources: - https://github.com/flyteorg/flyte/issues/5051 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 How does Admin send out input data for given flyte run. Is it thru wofklow CRD? Basically how the user data end up in propeller cluster so the execution can happen

FlyteAdmin sends input data for a workflow run by creating a FlyteWorkflow Custom Resource Definition (CRD) in the data plane (Propeller) cluster. The CRD contains references (not the raw data itself) to input data, which is stored in an object store (e.g., S3). Propeller then reads the CRD, retrieves the referenced input data from the object store, and launches the execution. This mechanism ensures that user data is passed securely and efficiently from the control plane to the data plane for execution, with only metadata and references included in the CRD, not the actual data payloads. For more details, see the architecture and data movement explanations in the and diagrams. Would you like a step-by-step breakdown of the data flow? Sources: - Figure - Figure Image-1 Image-2 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 so if inputs are going thru the Admin bucket (s3 on controlplane) why are outputs stored in local propeller-only accessed bucket? what sense it makes

Inputs are referenced by FlyteAdmin in the workflow CRD, but the actual input and output data are stored in object stores (S3 buckets) accessible to the data plane (Propeller). The control plane (Admin) typically stores only metadata and references, not the raw data itself. Outputs are written by Propeller to its own bucket because only the data plane needs to access them for downstream tasks or retrieval, and this design enforces security and isolation between clusters—Admin does not need access to all raw outputs, only to their references for tracking and lineage purposes. This separation is intentional: it allows each Propeller/data plane to manage its own data securely, while Admin coordinates workflow execution and metadata without direct access to all user data. For more, see the and clarifying notes in Flyte Issue #4683. Would you like a more detailed explanation of the security and architectural rationale? Sources: - Flyte Issue #4683 - Figure Image 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 in multicluster deployment, does propeller need access to my OIDC provider? I dont see it anywhere in the config

Yes, in a multicluster deployment with authentication enabled, each FlytePropeller instance (data plane) needs access to your OIDC provider to obtain tokens for authenticating with FlyteAdmin (control plane). This is required for secure communication, as Propeller uses client credentials or other OAuth2 flows to get access tokens. The necessary OIDC configuration (client ID, secret, endpoints) is typically provided via Kubernetes secrets and environment variables, not always directly visible in the main config files. For more details, see the discussion in Flyte GitHub Issue #2606 and Flyte GitHub Issue #2738. Would you like a step-by-step breakdown of how to configure Propeller for OIDC? Sources: - Flyte GitHub Issue #2606 - Flyte GitHub Issue #2738 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Yes, tell me where is the setting for propeller to set up path to OIDC and how to set up secret and client and all that jazz

For FlytePropeller to authenticate with FlyteAdmin using OIDC in a multicluster setup, you must provide the OIDC client ID and client secret via a Kubernetes Secret, and reference it in your Helm values. The actual OIDC endpoint (provider URL) is not set directly in the Propeller config; instead, Propeller uses the client credentials to obtain tokens and connect to Admin, which is configured with the OIDC provider details. 1. Create a Kubernetes Secret (e.g.,

flyte-secret-auth

) in the Propeller's namespace with the client secret:

apiVersion: v1
kind: Secret
metadata:
  name: flyte-secret-auth
  namespace: flyte
type: Opaque
stringData:
  client_secret: <YOUR_CLIENT_SECRET>

2. Set the client ID and secret in your Helm values (for flyte-core):

secrets:
  adminOauthClientCredentials:
    enabled: true
    clientSecret: "<YOUR_CLIENT_SECRET>"
    clientId: "flytepropeller"

This will mount the secret at

/etc/secrets/client_secret

in the Propeller pod. 3. Propeller config: The client ID and secret location are referenced in the Propeller config (see

admin.clientId

and

admin.clientSecretLocation

), but the OIDC base URL is only needed by Admin, not Propeller. For more, see the example in flyte-core values-keycloak-idp-flyteclients-without-browser.yaml and the discussion in Flyte Issue #2606. Would you like a step-by-step breakdown of the full setup? Sources: - flyte-core values-keycloak-idp-flyteclients-without-browser.yaml - Flyte Issue #2606 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 so if OIDC base url is only needed by admin, how does propeller get it, thru some HTTP communication?

Propeller does not need the OIDC base URL directly; it only needs the client ID and client secret to request tokens. When Propeller authenticates with FlyteAdmin, it uses these credentials to obtain tokens from the OIDC provider, but the actual OIDC configuration (base URL, endpoints) is managed by FlyteAdmin. Propeller relies on FlyteAdmin to provide the necessary authentication metadata during the token exchange and communication process. For more, see the discussion in Flyte Issue #2606. Would you like more detail on the authentication flow between Propeller and Admin? Sources: - Flyte Issue #2606 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 so propeller never talks directly with OIDC? It is just sending clientID and Secret into the admin?

Correct—Propeller does not talk directly to the OIDC provider. Instead, Propeller uses its client ID and client secret to authenticate with FlyteAdmin, which is responsible for handling the OIDC flow and token management. Propeller sends its credentials to Admin, and Admin manages the communication with the OIDC provider as needed. For more, see the explanation in Flyte Issue #2606. Would you like a step-by-step breakdown of this authentication flow? Sources: - Flyte Issue #2606 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.