salmon-refrigerator-32115
@task( container_image="<http://xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest|xyz.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest>", task_config=Spark( spark_conf={... } ), ) def read_spark_df() -> pandas.DataFrame: sess = flytekit.current_context().spark_session spark_df = sess.read.parquet("<s3a://bucket/key.parquet>").toPandas() ....
Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.