Hi, A question - how can I manually trigger an upl...
# ask-the-communityt
Tal
11/13/2022, 3:52 PMHi, A question - how can I manually trigger an upload (to s3 say) of a FlyteFile / FlyteDirectory?
k
Ketan (kumare3)
11/13/2022, 9:20 PM
cc @Yee I know we can give path, but not sure if we can manually trigger
Ketan (kumare3)
11/13/2022, 9:20 PM
@Tal why manually trigger?
t
Tal
11/13/2022, 9:28 PMUpload without the need to return a value (the flytefile/directory) some artifacts do note play a role in the downstream tasks of a workflow, so it is pointlaess to return them only for uploading
k
Ketan (kumare3)
11/13/2022, 9:30 PM
so the way i think about any generated artifact as being an output. Even if you do not want them to be used downstream, Flyte can automatically track it right? Why not produce it as an output and get the benefits
Ketan (kumare3)
11/13/2022, 9:30 PM
unless you want to hide that it is being produced?
Ketan (kumare3)
11/13/2022, 9:30 PM
if so, you will have to just trigger the upload on your own
t
Tal
11/13/2022, 9:38 PMI'm not sure hide is the way to put it... sometimes they are generated deep inside the code and returning them from the task could be somewhat annoying. If there was a tracking mechanism for artifacts that are not necessarily returned, but rather registered in some registry mechanism or something, it could be beneficial in some cases
Tal
11/13/2022, 9:40 PMCan I triger an upload on my own, and if so, can do it so it is recomnized in some way as a "signature" of the task?
k
Ketan (kumare3)
11/13/2022, 9:41 PM
interesting
Ketan (kumare3)
11/13/2022, 9:41 PM
this is fair
Ketan (kumare3)
11/13/2022, 9:43 PM
Flytekit does not have an automatic way to do that unfortunately. But writing up an issue would help in the future. Also “contributions :)“.
If I have to paraphrase, what you are suggesting is that it should be possible to create outputs without explicit return. Another solution can be you write a common output store and then return that
y
Yee
11/14/2022, 9:04 PM
kinda curious about the use-case here. can you describe what you’re doing?
Yee
11/14/2022, 9:04 PM
@Tal
Yee
11/14/2022, 9:05 PM
the notion of allowing users to upload file, and for flyte to keep track of them, is definitely something we’re thinking about.
Yee
11/14/2022, 9:06 PM
manually uploading of file right now is definitely not elegant
Yee
11/14/2022, 9:06 PM
here are some ways that you can do it
Yee
11/14/2022, 9:06 PM
Yee
11/14/2022, 9:07 PM
(caveat - i’ve been playing around with the flyte sandbox settings so yours might be a bit different)
Yee
11/14/2022, 9:08 PM
creating objects without an explicit return though is an interesting concept.
Yee
11/14/2022, 9:08 PM
yeah would love it if you could write up an issue. we’re specifically interested in the ux that you’d like to see.
t
Tal
11/14/2022, 9:55 PMThanks @Yee ! Thats helpful. Sure, I'll write up something. Making storage access seamless is important IMO. It could be by a context manager allowing to consume, manage and generate resources by name.
y
Yee
11/14/2022, 10:51 PM
yes, quite important. thank you. please don’t feel limited to just what we have right now though. feel free to think about what would work best for you.
Yee
11/14/2022, 10:51 PM
also if you could elaborate a bit more when you say “tracking mechanism for artifacts” - be helpful for us to understand how might that be used.
Yee
11/14/2022, 10:51 PM
thanks!
155 Views