#1895 Replaced user context spark session.
Pull request opened by
tionichm
TL;DR
These changes allows the pyspark.DataFrame SD transformer to obtain the Spark session after new_session was invoked.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
The solution is to get the spark session via the
SparkSession.builder.getOrCreate()
method call.
Tracking Issue
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
✅ All checks have passed
2/2 successful checks