How and where do I specify the location of a default remote place for all data regarding a project's executions? Things such as saved files should be stored to a specific S3 bucket by default

To specify a default remote location for storing data related to a project's executions in Flyte, you can configure the

raw-output-data-prefix

parameter. This parameter allows you to set a fully qualified storage path, such as an S3 bucket, where all data sandboxes should be stored. This can be done at the project or domain level, or even per execution. 1. Configuration at the Project/Domain Level: You can set the

raw-output-data-prefix

in the Flyte configuration files, such as

flytepropeller-config

, to define a default storage location for all executions within a project or domain. This is typically done by setting the

rawoutput-prefix

in the FlytePropeller configuration. 2. Execution Level Override: During workflow execution, you can override the default storage location by specifying a custom

raw-output-data-prefix

in the execution settings. This can be done through the Flyte UI or using command-line tools like

pyflyte

. For more details, refer to the Flyte documentation on data handling and FlytePropeller configuration. Would you like more detail on any specific part of this process? Sources: - Flyte documentation on data handling - FlytePropeller configuration 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Where do I put the propeller configuration? In the

.flyte/config-sandbox.yaml

?

It seems likely that the FlytePropeller configuration should be placed in a configuration file that is used by Flyte when deploying or running workflows. Typically, this configuration is part of the Kubernetes ConfigMap used by FlytePropeller. However, for local development or sandbox environments, you might use a file like

.flyte/config-sandbox.yaml

. 1. Kubernetes ConfigMap: In a production or Kubernetes environment, the FlytePropeller configuration is usually stored in a ConfigMap. This is referenced by the FlytePropeller deployment to configure its behavior. 2. Local/Sandbox Environment: For local development or sandbox environments, you can use a configuration file like

.flyte/config-sandbox.yaml

. This file can include settings for FlytePropeller, such as

rawoutput-prefix

and other relevant parameters. For more detailed instructions, you might want to refer to the Flyte documentation on running in a local cluster. Would you like more detail on setting up the configuration for different environments? Sources: - Flyte documentation on running in a local cluster 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.