Im currently running into alot of issues with spar...
# flyte-supportb
brainy-carpenter-31280
05/07/2025, 4:26 PMIm currently running into alot of issues with spark on flyte. Which integration works best for big data processing on flyte? Ray, Dask ...
a
agreeable-kitchen-44189
05/07/2025, 4:30 PMHi Julian!
What issues are you seeing?
Personally, i find the dask integration to work well (also in production settings) - disclaimer that I've authored it a while back.
b
brainy-carpenter-31280
05/07/2025, 4:31 PMI find dependency resolution very hard. Basically outputting a spark dataframe does not work for me currently. Ill try dask
a
agreeable-kitchen-44189
05/07/2025, 4:32 PMFeel free to reach out in case you have any questions!
f
freezing-airport-6809
05/07/2025, 11:55 PM
Hmm @brainy-carpenter-31280 that’s odd
freezing-airport-6809
05/07/2025, 11:55 PM
Can you share an example
freezing-airport-6809
05/08/2025, 12:42 AM
We have large scale deployments of Spark as well , so would love to learn problems
b
brainy-carpenter-31280
05/08/2025, 6:08 AMhttps://github.com/julianStreibel/flyte-spark
This is a minimal example. I added jars fixing this particular issue (see readme). But its cascading.
brainy-carpenter-31280
05/08/2025, 6:10 AMWhat is your spark setup? Are you using custom images with pre loaded dependencies? How are you caching dependencies in the cluster? Do you have a shared pvc?
f
freezing-airport-6809
05/08/2025, 11:22 AM
Maybe this repo is private
freezing-airport-6809
05/08/2025, 11:23 AM
Yes using custom images - why donii to oh need a pvc - why not build a new image
b
brainy-carpenter-31280
05/08/2025, 11:26 AMyes sorry its public now
f
freezing-airport-6809
05/08/2025, 11:27 AM
Is this not working ?
b
brainy-carpenter-31280
05/08/2025, 11:27 AMyes raises the exception in the readme
f
freezing-airport-6809
05/08/2025, 11:28 AM
One problem I see is your cpu/mem is too low - spark is hungry
b
brainy-carpenter-31280
05/08/2025, 11:28 AMI know but that is not the problem here
f
freezing-airport-6809
05/08/2025, 11:28 AM
I am AFK, will try later today and share the output
b
brainy-carpenter-31280
05/08/2025, 11:28 AMThanks 🙂
f
freezing-airport-6809
05/08/2025, 7:09 PM
@brainy-carpenter-31280 it works
freezing-airport-6809
05/08/2025, 7:09 PM
I did change the image a bit
Copy code
custom_image = ImageSpec(
- registry="<http://211125663991.dkr.ecr.us-west-2.amazonaws.com|211125663991.dkr.ecr.us-west-2.amazonaws.com>",
- name="flytekit",
+ # registry="<http://211125663991.dkr.ecr.us-west-2.amazonaws.com|211125663991.dkr.ecr.us-west-2.amazonaws.com>",
+ registry="<http://ghcr.io/unionai-oss|ghcr.io/unionai-oss>",
+ name="spark",
packages=[
# Needed for IPv6 support
"flytekit",
"flytekitplugins-spark",
- "pyspark==3.4.0",
+ # "pyspark==3.4.0",
],
- pip_extra_args="--pre",
- platform="linux/arm64",
+ # pip_extra_args="--pre",
+ # platform="linux/arm64",
)freezing-airport-6809
05/08/2025, 7:10 PM
not much, i removed pyspark 3.4.0
freezing-airport-6809
05/08/2025, 7:10 PM
flytekitplugins brings spark
freezing-airport-6809
05/08/2025, 7:10 PM
but i can try with 3.4.0 too
freezing-airport-6809
05/08/2025, 7:13 PM
trying with 3.4.0
freezing-airport-6809
05/08/2025, 7:35 PM
it must be arm
b
brainy-carpenter-31280
05/14/2025, 5:10 PMThanks for testing!
I currently only have arm nodes in the cluster :/
brainy-carpenter-31280
05/14/2025, 5:16 PMI have the same problem with reading StructuredDatasets. Can I somehow read in the s3 uri and just read parquets from there? I tried returning a pandas dataframe in a structuredDataset and read it in the spark task as a structured dataset reading it from the uri but it is None.