Hi, A question - how can I manually trigger an upload (to s3 say) of a FlyteFile / FlyteDirectory?
11/13/2022, 9:20 PM
cc @Yee I know we can give path, but not sure if we can manually trigger
@Tal why manually trigger?
11/13/2022, 9:28 PM
Upload without the need to return a value (the flytefile/directory) some artifacts do note play a role in the downstream tasks of a workflow, so it is pointlaess to return them only for uploading
11/13/2022, 9:30 PM
so the way i think about any generated artifact as being an output. Even if you do not want them to be used downstream, Flyte can automatically track it right? Why not produce it as an output and get the benefits
unless you want to hide that it is being produced?
if so, you will have to just trigger the upload on your own
11/13/2022, 9:38 PM
I'm not sure hide is the way to put it... sometimes they are generated deep inside the code and returning them from the task could be somewhat annoying. If there was a tracking mechanism for artifacts that are not necessarily returned, but rather registered in some registry mechanism or something, it could be beneficial in some cases
Can I triger an upload on my own, and if so, can do it so it is recomnized in some way as a "signature" of the task?
11/13/2022, 9:41 PM
this is fair
Flytekit does not have an automatic way to do that unfortunately. But writing up an issue would help in the future. Also “contributions :)“.
If I have to paraphrase, what you are suggesting is that it should be possible to create outputs without explicit return. Another solution can be you write a common output store and then return that
11/14/2022, 9:04 PM
kinda curious about the use-case here. can you describe what you’re doing?
the notion of allowing users to upload file, and for flyte to keep track of them, is definitely something we’re thinking about.
manually uploading of file right now is definitely not elegant
(caveat - i’ve been playing around with the flyte sandbox settings so yours might be a bit different)
creating objects without an explicit return though is an interesting concept.
yeah would love it if you could write up an issue. we’re specifically interested in the ux that you’d like to see.
11/14/2022, 9:55 PM
Thanks @Yee ! Thats helpful. Sure, I'll write up something. Making storage access seamless is important IMO. It could be by a context manager allowing to consume, manage and generate resources by name.
11/14/2022, 10:51 PM
yes, quite important. thank you. please don’t feel limited to just what we have right now though. feel free to think about what would work best for you.
also if you could elaborate a bit more when you say “tracking mechanism for artifacts” - be helpful for us to understand how might that be used.