#3350 fix: update Spark plugin for compatibility with Spark 3.x and 4.x Pull request opened by kevinliao852 ## Tracking issue ## Why are the changes needed? The current Spark plugin imports and uses Spark 4-specific data types that do not exist in Spark 3.x (including Spark 3.4). This results in runtime import errors as

File "/home/kevinliao/.local/share/uv/python/cpython-3.10.17-linux-x86_64-gnu/lib/python3.10/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'pyspark.sql.classic'

Because Flyte users run Spark workloads across mixed versions (Spark 3.x or Spark 4.x), the plugin must not assume Spark 4 APIs exist at runtime. Without this patch, Spark 3.x tasks fail immediately, even if their logic does not depend on Spark-4-only features. ## What changes were proposed in this pull request? 1. Added runtime checks for Spark major version. 2. Implemented conditional import of Spark 4 data types. ## How was this patch tested? Tested PySpark locally with Spark 3.4 and Spark 4.x. Built a Flyte sandbox image with the updated transformer and schema. Ran Spark tasks in Flyte using: Spark 3.4 base image → passed Spark 4.x base image → passed ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito • This pull request enhances the Flyte Spark plugin to ensure compatibility with both Spark 3.x and 4.x by implementing runtime checks for Spark major versions. • It introduces conditional imports for Spark 4-specific data types to prevent runtime errors when using Spark 3.x. • The modifications include the registration of encoding and decoding handlers that adapt to the available Spark version. • Overall, this pull request touches on the Flyte Spark plugin, enhancing compatibility across Spark versions and introducing runtime checks to prevent errors. flyteorg/flytekit