Greg Gydush03/17/2022, 2:49 AM
, we end up returning something like this when we want to upload a file to specific GCS path. This is fine if you have one file you are returning, but can be verbose when working with a handful of output files.
Does anyone have recommendations to simplify this? I was thinking a small PR adding an additional parameter to FlyteFile could be nice, but thought I’d ask here in case anyone has other ideas to simplify!
FlyteFile(random_local_path, remote_path=os.path.join(gcs_outdir, os.path.basename(random_local_path))
FlyteFile(os.path.join(current_context().working_directory, "message.txt"), remote_path=....)
Greg Gydush03/17/2022, 1:34 PM
why don’t you set the raw output prefix?Could you maybe elaborate on this too? This can be set on the context?
Greg Gydush03/21/2022, 12:52 AM
Greg Gydush03/21/2022, 12:55 AM
is fine. Does that make sense? I can show you some examples of why I would need something like that (it’s extremely common in bioinfo — e.g., intermediate files upload to ephemeral location so next task can use, but keep the logs in permanent GCS location)
would be handy, but may be best to discuss more in OH