Hi team, is there a way to configure the s3 endpoint url. We’re looking to use interface endpoint to access s3. An example endpoint would look like: vpce-aaaabbbbcccc-dddd.vpce-svc-12345678.ap-east-1.vpce.amazonaws.com https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#vpce-private-dns:~:text=The%20D[…]ts%20are%20publicly%20resolvable

this feels a bit beyond the scope of flyte… but could you elaborate a bit please?

To access s3, awscli supports specifying endpoint:

aws --region ap-east-1 --endpoint-url <https://bucket.vpce-0bc61e1834692b45c-4re8q9i4.s3.ap-east-1.vpce.amazonaws.com> s3 ls s3://<bucket_name>

This will optimize our use case as this endpoint allows us to access s3 via VPC rather than through public internet. We’d like to specify endpoint url as

<https://bucket.vpce-0bc61e1834692b45c-4re8q9i4.s3.ap-east-1.vpce.amazonaws.com>

for Flyte to use. I can see Stow supports this config (https://github.com/graymeta/stow/blob/master/s3/config.go#L43), but I’m not sure if flyte supports it and how I should specify this config in the charts.

yes, you can change the endpoint in default env. https://github.com/flyteorg/flyte/blob/f94f8830815318af907874a9cab1bf54f371f2be/charts/flyte-core/values-sandbox.yaml#L74

Thanks Kevin! I have updated the endpoint at this place. Just want to confirm that this is the only place that I need to modify. The reason I asked is that when I generate the final helm chart to be deployed, I’m seeing several places where bucket is specified but endpoint is not. For example: 1. flyte-admin-base-config | storage.yaml 2. flyte-console-config | storage.yaml 3. flyte-propeller-config | storage.yaml

if you only want to change endpoint for flytekit, only need to update default env above. if propeller and admin also need to write data to different endpoint, then need to update propeller and admin config

How do know the what configs are available on admin and propeller? At least on the official values.yaml, I don’t see where I can config the endpoints

@Kevin Su revisit please!

in the value.yaml, by default, both admin and propeller will use the same config, so you have to manually change the config map in the k8s