Hi, everyone. I have a naive question: How do I load a file from a GCS bucket inside a spark_task? Is there some idiomatic ways to do it in Flyte? (Normally I would add some maven coordinates to my spark config, but I noticed that this does not work for Flyte in a cluster where spark images has to be pre-built.)

In Flyte a task can use file type for input/output, Flyte uses s3/gcs for persistent storage. If that file return by a task in flyte then you can pass that file to your spark task as input, You don’t need to deal with anything. If that file is not generated return by flyte, Then you need to pass the file path url in task and task will use gcs client to download the file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/files.html

so if I understand it correctly, I should be able to read a gcs file in flyte like this:

@task
def get_file(url) -> FlyteFile:
    return FlyteFile(url)

@task(task_config=Spark(...))
def spark_task(file: FlyteFile):
    spark = flytekit.current_session().spark_session
    df = spark.read.text(file)

@workflow
def pipeline(url="<gs://xxxx>"):
    file = get_file(url)
    spark_task(file)

Is this the right way to do it?

yes

Do i need to an one more step for FlyteFile(url).download()

no 😁

That sounds great! Thx a lot.

sorry my bad, You need

file.download()

, @Samhita Alla Can you confirm the logic ?

Oh yes.

file.download()

is required.

So the handle returned from

file.download()

should be feed into

spark.read.text(..)

right?

Check one example https://github.com/flyteorg/flytesnacks/blob/master/cookbook/core/flyte_basics/files.py#L40

Also docs on internals here https://docs.flyte.org/en/latest/concepts/data_management.html#divedeep-data-management

Yes, but let me be clear here: what do you mean by a remote_path, if not the original url? But if you dont have a gcs_connector on your executor, then how does spark read it anyway, so I dont get the point of wrapping my url inside a FlyteFile then return it back again.

@Tiansu Yu I can probably help - but I am in PDT, so in 2 ish hours

Thx for these points.

Definitely read the docs on both