#3630 [BUG] K8s Spark plugin can't read or write from s3 out of the box Issue created by peridotml Describe the bug A few people (including me) have run into the following error when trying to save a DataFrame from a spark task when using s3 as a file system.

error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o42.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

• https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to. • https://discuss.flyte.org/t/39240/U03CZJFCT7T-you-should-be-able-to-find-pods-in-the-flytesnac Expected behavior We would expect this to work out of the box by providing sensible defaults. I did find a solution. The task works when I include

configure_s3

. Additional context to reproduce

import pandas as pd

from flytekit import task, workflow, StructuredDataset
from flytekitplugins.spark import Spark
import flytekit

def configure_s3(spark):
    hadoop_conf = spark._jsc.hadoopConfiguration()
    hadoop_conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    hadoop_conf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.WebIdentityTokenCredentialsProvider")


@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:k8s_spark-latest|ghcr.io/flyteorg/flytecookbook:k8s_spark-latest>", task_config=Spark(
        spark_conf={
            "spark.driver.memory": "1000M",
            "spark.executor.memory": "1000M",
            "spark.executor.instances": "3",
            "spark.driver.cores": "1",
            "spark.executor.cores": "1",
        }),
)
def t1() -> StructuredDataset:
    spark = flytekit.current_context().spark_session
    
    # The task fails without these settings.
    # configure_s3(spark)

    df = pd.DataFrame({"a": [1,2,3]})
    df = spark.createDataFrame(df)
    return StructuredDataset(dataframe=df)

@workflow
def wf() -> StructuredDataset:
    sd = t1()
    return sd

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte