Hello,
I have upstream data from databricks DBFS t...
# ask-the-communityf
Frank Shen
02/15/2024, 10:34 PMHello,
I have upstream data from databricks DBFS tables that I would like flyte tasks to consume from.
Is this supported by Flyte?
What are some of the approaches you suggest?
Thanks.
k
Ketan (kumare3)
02/16/2024, 1:06 AM
is this just input? Can you read it using some python library?
Ketan (kumare3)
02/16/2024, 1:07 AM
if so you can make a transformer to read from DBFs ? HDFS
Ketan (kumare3)
02/16/2024, 1:07 AM
also flytes fsspec supports hdfs already - https://github.com/fsspec/filesystem_spec/issues/663
Ketan (kumare3)
02/16/2024, 1:08 AM
BTW, if you want to connect upstream with downstream, this is possible now with Union Artifacts
Ketan (kumare3)
02/16/2024, 1:08 AM
happy to do a demo
f
Frank Shen
02/16/2024, 1:18 AMThanks @Ketan (kumare3), I will look into it.
7 Views