<#3272 Add support pyspark.sql.classic.dataframe.D...
# flyte-github
c
#3272 Add support pyspark.sql.classic.dataframe.DataFrame transformer Pull request opened by arbaobao ## Tracking issue #6478 ## Why are the changes needed? There is a new type of dataframe when `pyspark`>=4.0.0. It's not recognized by the current structured dataset. Therefore, the type transformer will fail to serialize and deserialize it. ## What changes were proposed in this pull request? Add a new type of spark dataframe and register it. ## How was this patch tested? Github CI will fail if we don't add the new type of dataframe and pyspark>=4.0.0. ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link https://bito.ai#summarystart| ### Summary by Bito This pull request adds support for the new
pyspark.sql.classic.dataframe.DataFrame
type for compatibility with
pyspark
version 4.0.0 and above. It includes new classes for reading and writing this DataFrame type, along with serialization and deserialization handlers. Plugin requirements have also been updated accordingly. flyteorg/flytekit