Greg Gydush
03/17/2022, 2:49 AMFlyteFile
, we end up returning something like this when we want to upload a file to specific GCS path. This is fine if you have one file you are returning, but can be verbose when working with a handful of output files.
FlyteFile(random_local_path, remote_path=os.path.join(gcs_outdir, os.path.basename(random_local_path))
Does anyone have recommendations to simplify this? I was thinking a small PR adding an additional parameter to FlyteFile could be nice, but thought I’d ask here in case anyone has other ideas to simplify!
FlyteFile(random_local_path, remote_dir=gcs_outdir)
FlyteFile(os.path.join(current_context().working_directory, "message.txt"), remote_path=....)
Yee
Greg Gydush
03/17/2022, 1:34 PMwhy don’t you set the raw output prefix?Could you maybe elaborate on this too? This can be set on the context?
Yee
Greg Gydush
03/21/2022, 12:52 AMYee
Greg Gydush
03/21/2022, 12:55 AMraw_data_prefix
is fine. Does that make sense? I can show you some examples of why I would need something like that (it’s extremely common in bioinfo — e.g., intermediate files upload to ephemeral location so next task can use, but keep the logs in permanent GCS location)remote_dir
would be handy, but may be best to discuss more in OHYee