@ancient-wolf-19325 Can I configure a different s3 bucket per project?

Yes, you can configure a different S3 bucket per project in Flyte. This can be achieved by specifying the

raw_output_data_config

as part of a

LaunchPlan

. You can update the workflow execution configuration to set the

output_location_prefix

for each project.

domain: development
project: flytesnacks
raw_output_data_config:
  output_location_prefix: s3://my-s3-bucket

For more details, refer to the Flyte documentation. Would you like more detail on this configuration? Sources: - Flyte Documentation - Flyte Community Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Can I configure a separate S3 bucket for inputs and outputs to workflows?

Yes, you can configure separate S3 buckets for inputs and outputs in Flyte. This can be done by specifying different storage configurations for metadata and user data in the Flyte deployment configuration. For example, in the Flyte core chart, you can specify the

metadataContainer

for Flyte metadata and the

userDataContainer

for user data:

storage:
  metadataContainer: "s3://my-metadata-bucket"
  userDataContainer: "s3://my-user-data-bucket"

This configuration ensures that Flyte metadata and user data are stored in separate S3 buckets. Would you like more detail on this configuration? Sources: - Flyte GitHub Issue #4291 - Flyte Community Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Can I configure a separate S3 bucket for inputs and outputs to workflows for each project?

Yes, you can configure separate S3 buckets for inputs and outputs for each project in Flyte. This can be achieved by specifying different

raw_output_data_config

settings in the

LaunchPlan

for each project. For example, you can set the

output_location_prefix

for outputs and configure the input storage separately in the Flyte configuration:

domain: development
project: flytesnacks
raw_output_data_config:
  output_location_prefix: s3://my-output-bucket

For inputs, you can configure the FlyteAdmin to generate signed URLs for different buckets based on the context. This can be done by modifying the FlyteAdmin storage configuration to differentiate between metadata and data buckets. Would you like more detail on this configuration? Sources: - Flyte Documentation - Flyte Community Discussion - Flyte GitHub Issue #4291 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@bumpy-match-83743 could you confirm if some of the above answers work for you? I've been also exploring the extent to which data placement can be customtized

I haven’t got to this point in my project yet, but will be there next week. If you’re suggesting that we cannot configure as we buckets per tenant, we may have to abandon altogether as it’s an important security concern.

yep, the workflow execution config should do the trick for that. If you find anything different please let us know