Hi, A question - how can I manually trigger an upl...
# flyte-supporth
high-kitchen-90454
11/13/2022, 3:52 PMHi, A question - how can I manually trigger an upload (to s3 say) of a FlyteFile / FlyteDirectory?
f
freezing-airport-6809
11/13/2022, 9:20 PM
cc @thankful-minister-83577 I know we can give path, but not sure if we can manually trigger
freezing-airport-6809
11/13/2022, 9:20 PM
@high-kitchen-90454 why manually trigger?
h
high-kitchen-90454
11/13/2022, 9:28 PMUpload without the need to return a value (the flytefile/directory) some artifacts do note play a role in the downstream tasks of a workflow, so it is pointlaess to return them only for uploading
f
freezing-airport-6809
11/13/2022, 9:30 PM
so the way i think about any generated artifact as being an output. Even if you do not want them to be used downstream, Flyte can automatically track it right? Why not produce it as an output and get the benefits
freezing-airport-6809
11/13/2022, 9:30 PM
unless you want to hide that it is being produced?
freezing-airport-6809
11/13/2022, 9:30 PM
if so, you will have to just trigger the upload on your own
h
high-kitchen-90454
11/13/2022, 9:38 PMI'm not sure hide is the way to put it... sometimes they are generated deep inside the code and returning them from the task could be somewhat annoying. If there was a tracking mechanism for artifacts that are not necessarily returned, but rather registered in some registry mechanism or something, it could be beneficial in some cases
high-kitchen-90454
11/13/2022, 9:40 PMCan I triger an upload on my own, and if so, can do it so it is recomnized in some way as a "signature" of the task?
f
freezing-airport-6809
11/13/2022, 9:41 PM
interesting
freezing-airport-6809
11/13/2022, 9:41 PM
this is fair
freezing-airport-6809
11/13/2022, 9:43 PM
Flytekit does not have an automatic way to do that unfortunately. But writing up an issue would help in the future. Also “contributions :)“.
If I have to paraphrase, what you are suggesting is that it should be possible to create outputs without explicit return. Another solution can be you write a common output store and then return that
👍 1
t
thankful-minister-83577
11/14/2022, 9:04 PM
kinda curious about the use-case here. can you describe what you’re doing?
thankful-minister-83577
11/14/2022, 9:04 PM
@high-kitchen-90454
thankful-minister-83577
11/14/2022, 9:05 PM
the notion of allowing users to upload file, and for flyte to keep track of them, is definitely something we’re thinking about.
thankful-minister-83577
11/14/2022, 9:06 PM
manually uploading of file right now is definitely not elegant
thankful-minister-83577
11/14/2022, 9:06 PM
here are some ways that you can do it
thankful-minister-83577
11/14/2022, 9:06 PM
Untitled.py
thankful-minister-83577
11/14/2022, 9:07 PM
(caveat - i’ve been playing around with the flyte sandbox settings so yours might be a bit different)
thankful-minister-83577
11/14/2022, 9:08 PM
creating objects without an explicit return though is an interesting concept.
thankful-minister-83577
11/14/2022, 9:08 PM
yeah would love it if you could write up an issue. we’re specifically interested in the ux that you’d like to see.
h
high-kitchen-90454
11/14/2022, 9:55 PMThanks @thankful-minister-83577 ! Thats helpful. Sure, I'll write up something. Making storage access seamless is important IMO. It could be by a context manager allowing to consume, manage and generate resources by name.
t
thankful-minister-83577
11/14/2022, 10:51 PM
yes, quite important. thank you. please don’t feel limited to just what we have right now though. feel free to think about what would work best for you.
thankful-minister-83577
11/14/2022, 10:51 PM
also if you could elaborate a bit more when you say “tracking mechanism for artifacts” - be helpful for us to understand how might that be used.
thankful-minister-83577
11/14/2022, 10:51 PM
thanks!
156 Views