https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Frank Shen

02/15/2024, 10:34 PM
Hello, I have upstream data from databricks DBFS tables that I would like flyte tasks to consume from. Is this supported by Flyte? What are some of the approaches you suggest? Thanks.
k

Ketan (kumare3)

02/16/2024, 1:06 AM
is this just input? Can you read it using some python library?
if so you can make a transformer to read from DBFs ? HDFS
also flytes fsspec supports hdfs already - https://github.com/fsspec/filesystem_spec/issues/663
BTW, if you want to connect upstream with downstream, this is possible now with Union Artifacts
happy to do a demo
f

Frank Shen

02/16/2024, 1:18 AM
Thanks @Ketan (kumare3), I will look into it.