Chris Grass
10/25/2023, 4:08 PMRemoteURLInterface
for Azure (example). It seems the interface is only called when remoteDataConfig.SignedURL.Enabled == true
, is that right?
Also, I noticed that the aws implementation uses the s3 client directly to fetch a signed URL from a created Request object which does not appear to be used (beyond fetching the URL).
I found that surprising; does anyone know why RawStore.CreateSignedURL
isn't used for that purpose? It feels like offloading that responsibility to the store implementation (likely Stow) would work, while avoiding having to re-implement the logic in each concrete RemoteURL type.Yee
Chris Grass
10/30/2023, 2:40 PMas for the url that’s returned however, that is used. this is part of the fast-registration/upload of offloaded data types flow used in flytekit.ok, that makes sense. but the URL returned from s3 client and the URL returned from stow should be the same, right? i think stow has an s3 implementation, which would likely use the s3 client in the same manner as the flyte code
as for the url that’s returned however, that is used. this is part of the fast-registration/upload of offloaded data types flow used in flytekit.right, the URL is used to fetch data from the store. but only when
remoteDataConfig.SignedURL.Enabled == true
, correct?Yee
Chris Grass
11/30/2023, 2:55 PM