Hi A question how can I manually trigger an upload to s3 say

Question

Hi A question how can I manually trigger an upload to s3 say of a FlyteFile FlyteDirectory

Ketan (kumare3) · Answer

cc < Yee> I know we can give path but not sure if we can manually trigger

Ketan (kumare3) · Answer

< Tal> why manually trigger

Tal · Answer

Upload without the need to return a value the flytefile directory some artifacts do note play a role in the downstream tasks of a workflow so it is pointlaess to return them only for uploading

Ketan (kumare3) · Answer

so the way i think about any generated artifact as being an output Even if you do not want them to be used downstream Flyte can automatically track it right Why not produce it as an output and get the benefits

Ketan (kumare3) · Answer

unless you want to hide that it is being produced

Ketan (kumare3) · Answer

if so you will have to just trigger the upload on your own

Tal · Answer

I m not sure hide is the way to put it sometimes they are generated deep inside the code and returning them from the task could be somewhat annoying If there was a tracking mechanism for artifacts that are not necessarily returned but rather registered in some registry mechanism or something it could be beneficial in some cases

Tal · Answer

Can I triger an upload on my own and if so can do it so it is recomnized in some way as a signature of the task

Ketan (kumare3) · Answer

interesting

Ketan (kumare3) · Answer

this is fair

Ketan (kumare3) · Answer

Flytekit does not have an automatic way to do that unfortunately But writing up an issue would help in the future Also contributions If I have to paraphrase what you are suggesting is that it should be possible to create outputs without explicit return Another solution can be you write a common output store and then return that

Yee · Answer

kinda curious about the use case here can you describe what you re doing

Yee · Answer

< Tal>

Yee · Answer

the notion of allowing users to upload file and for flyte to keep track of them is definitely something we re thinking about

Yee · Answer

manually uploading of file right now is definitely not elegant

Yee · Answer

here are some ways that you can do it

Yee · Answer

caveat i ve been playing around with the flyte sandbox settings so yours might be a bit different

Yee · Answer

creating objects without an explicit return though is an interesting concept

Yee · Answer

yeah would love it if you could write up an issue we re specifically interested in the ux that you d like to see

Tal · Answer

Thanks < Yee> Thats helpful Sure I ll write up something Making storage access seamless is important IMO It could be by a context manager allowing to consume manage and generate resources by name

Yee · Answer

yes quite important thank you please don t feel limited to just what we have right now though feel free to think about what would work best for you

Yee · Answer

also if you could elaborate a bit more when you say tracking mechanism for artifacts be helpful for us to understand how might that be used

Yee · Answer

thanks