Hello,
I have upstream data from databricks DBFS t...
# flyte-supports
salmon-refrigerator-32115
02/15/2024, 10:34 PMHello,
I have upstream data from databricks DBFS tables that I would like flyte tasks to consume from.
Is this supported by Flyte?
What are some of the approaches you suggest?
Thanks.
f
freezing-airport-6809
02/16/2024, 1:06 AM
is this just input? Can you read it using some python library?
freezing-airport-6809
02/16/2024, 1:07 AM
if so you can make a transformer to read from DBFs ? HDFS
freezing-airport-6809
02/16/2024, 1:07 AM
also flytes fsspec supports hdfs already - https://github.com/fsspec/filesystem_spec/issues/663
freezing-airport-6809
02/16/2024, 1:08 AM
BTW, if you want to connect upstream with downstream, this is possible now with Union Artifacts
freezing-airport-6809
02/16/2024, 1:08 AM
happy to do a demo
s
salmon-refrigerator-32115
02/16/2024, 1:18 AMThanks @freezing-airport-6809, I will look into it.
8 Views