https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
d

Debajyoti Chatterjee

02/19/2024, 9:40 AM
Deployment context: https://flyte-org.slack.com/archives/C01P3B761A6/p1708334025321359 I noticed that the workflow inputs are stored in the Bucket configured in control plane components. Apart from this, it is also inlined in the Workflow manifest YAML,
Copy code
inputs:
  literals:
    tokenizer:
      scalar:
        primitive:
          stringValue: a
Is there any way we can ask the control plane components to not keep the actual values in the control plane bucket?
d

David Espejo (he/him)

02/19/2024, 4:13 PM
AFAICT and considering the docs I'd say that the metadata store (where Inputs are persisted) is shared by both the control and data planes Regardless of the Helm chart you're using, you can specify separate buckets for metadata and Raw data but not a metadata bucket per cluster
d

Debajyoti Chatterjee

02/20/2024, 3:59 AM
Regardless of the Helm chart you're using, you can specify separate buckets for metadata and Raw data but not a metadata bucket per cluster (
Yes, that's what I figured from the chart.
the metadata store (where Inputs are persisted) is shared by both the control and data planes
What I observed is that the data plane can work without access to the metadata bucket (control plane), and this is because the inputs are already bound in the Workflow manifest.
d

David Espejo (he/him)

02/20/2024, 2:59 PM
Right. Compiled protobuf representation of the workflow structure is stored in the database from where propeller will retrieve it to start an execution, so in theory it enables two different clusters to not share the metadata bucket Can you please try enabling multi-container? Add this to your Helm chart config and then update:
Copy code
storage:
    enableMultiContainer: true
3 Views