https://flyte.org logo
#ask-the-community
Title
# ask-the-community
k

Kamakshi Muthukrishnan

03/09/2023, 9:44 AM
@Kevin Su @Samhita Alla @Ketan (kumare3) - We are getting the below error while executing Spark task. Could you please help us An error occurred while calling o125.collectToPython. : java.lang.OutOfMemoryError: Java heap space at org.apache.spark.sql.execution.SparkPlan$$anon$1._next(SparkPlan.scala:391) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:402) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:388) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1(SparkPlan.scala:425) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1$adapted(SparkPlan.scala:424) at org.apache.spark.sql.execution.SparkPlan$$Lambda$2872/0x0000000841442840.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:424) at org.apache.spark.sql.Dataset.$anonfun$collectToPython$1(Dataset.scala:3688) at org.apache.spark.sql.Dataset$$Lambda$2466/0x0000000841284840.apply(Unknown Source) at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3858) at org.apache.spark.sql.Dataset$$Lambda$2735/0x00000008413bdc40.apply(Unknown Source) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3856) at org.apache.spark.sql.Dataset$$Lambda$2467/0x0000000841285440.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2475/0x0000000841294840.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2468/0x0000000841290040.apply(Unknown Source) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3856) at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3685) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) User error.
e

Evan Sadler

03/09/2023, 1:31 PM
This looks like an out of memory error. Usually increasing is the driver and worker memory helps as a first step. However, if you are collecting a spark dataframe into python or pandas, you might want to set
spark.driver.maxResultSize=0
This allows larger amounts of data to move into python. I think you work at Warner media, so feel free to share the code with me. Probably the best way to debug this.
k

Kamakshi Muthukrishnan

03/09/2023, 1:39 PM
@Evan Sadler - This is already set..
e

Evan Sadler

03/09/2023, 2:36 PM
Increasing the driver memory from 30Gi to 40Gi fixed the issue 🎉
k

Ketan (kumare3)

03/09/2023, 3:17 PM
@Kamakshi Muthukrishnan may I respectfully ask not to tag people in the community. This is really community help, We are looking at many issues and using our free time to answer. In this case the issue is clearly with spark. In the past it has been ray, I would recommend please work with those communities- these are not Flyte problems. Spark and ray are hard to use systems and we are not experts. We will try our best but please do not tag people
Also I am sorry to remind you this Samhita, Kevin and I helped you in the past and if you get a problem you turn around and tag us immediately. I want to be highly respectful and helpful, but all these folks have full time jobs. Supporting the community is altruistic by them. Please consider this
e

Evan Sadler

03/09/2023, 4:14 PM
@Ketan (kumare3) I appreciate your feedback and I am taking your comments very seriously. We are very thankful for all the help you and Union have provided the past year. You have all been so generous with your time and we really want to respect it going forward. I started a slack thread with the teams using Flyte at warner brothers / discovery. We are going to make sure that 1) folks know to not tag Union employees and 2) we debug over in WBD slack before elevating to the Flyte channel.
k

Ketan (kumare3)

03/09/2023, 4:24 PM
Highly appreciated @Evan Sadler - please do not get the wrong signal
We want the community to be scalabale
f

Frank Shen

03/09/2023, 5:01 PM
@Ketan (kumare3), thanks for bringing this issue to our attention. We will educate people new to this community.
63 Views