#4247 [BUG] Spark plugin new_session function causes the pyspark.sql.DataFrame SD transformer decode to break Issue created by tionichm Describe the bug After invoking

flytekitplugins.spark.new_session(...)

, the

pyspark.sql.DataFrame

SD transformer's decoder breaks. The reason for this is due to it referencing

user_ctx.spark_session

that was set during the Spark tasks'

pre_execute

method call. Expected behavior The

pyspark.sql.DaraFrame

SD transformer should use whichever spark session is active, and should be decoupled from the flyte context. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? • Yes Have you read the Code of Conduct? • Yes flyteorg/flyte