@Kevin Su @Samhita Alla @Ketan (kumare3) - We are getting the below error while executing Spark task. Could you please help us An error occurred while calling o125.collectToPython. : java.lang.OutOfMemoryError: Java heap space at org.apache.spark.sql.execution.SparkPlan$$anon$1._next(SparkPlan.scala:391) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:402) at org.apache.spark.sql.execution.SparkPlan$$anon$1.getNext(SparkPlan.scala:388) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1(SparkPlan.scala:425) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeCollect$1$adapted(SparkPlan.scala:424) at org.apache.spark.sql.execution.SparkPlan$$Lambda$2872/0x0000000841442840.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:424) at org.apache.spark.sql.Dataset.$anonfun$collectToPython$1(Dataset.scala:3688) at org.apache.spark.sql.Dataset$$Lambda$2466/0x0000000841284840.apply(Unknown Source) at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:3858) at org.apache.spark.sql.Dataset$$Lambda$2735/0x00000008413bdc40.apply(Unknown Source) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3856) at org.apache.spark.sql.Dataset$$Lambda$2467/0x0000000841285440.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2475/0x0000000841294840.apply(Unknown Source) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95) at org.apache.spark.sql.execution.SQLExecution$$$Lambda$2468/0x0000000841290040.apply(Unknown Source) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3856) at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3685) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) User error.

This looks like an out of memory error. Usually increasing is the driver and worker memory helps as a first step. However, if you are collecting a spark dataframe into python or pandas, you might want to set

spark.driver.maxResultSize=0

This allows larger amounts of data to move into python. I think you work at Warner media, so feel free to share the code with me. Probably the best way to debug this.

@Evan Sadler - This is already set..

Increasing the driver memory from 30Gi to 40Gi fixed the issue 🎉

@Kamakshi Muthukrishnan may I respectfully ask not to tag people in the community. This is really community help, We are looking at many issues and using our free time to answer. In this case the issue is clearly with spark. In the past it has been ray, I would recommend please work with those communities- these are not Flyte problems. Spark and ray are hard to use systems and we are not experts. We will try our best but please do not tag people

@Ketan (kumare3) I appreciate your feedback and I am taking your comments very seriously. We are very thankful for all the help you and Union have provided the past year. You have all been so generous with your time and we really want to respect it going forward. I started a slack thread with the teams using Flyte at warner brothers / discovery. We are going to make sure that 1) folks know to not tag Union employees and 2) we debug over in WBD slack before elevating to the Flyte channel.

Highly appreciated @Evan Sadler - please do not get the wrong signal

@Ketan (kumare3), thanks for bringing this issue to our attention. We will educate people new to this community.