#3272 Add support pyspark.sql.classic.dataframe.DataFrame transformer
Pull request opened by
arbaobao
## Tracking issue
#6478
## Why are the changes needed?
There is a new type of dataframe when `pyspark`>=4.0.0. It's not recognized by the current structured dataset. Therefore, the type transformer will fail to serialize and deserialize it.
## What changes were proposed in this pull request?
Add a new type of spark dataframe and register it.
## How was this patch tested?
Github CI will fail if we don't add the new type of dataframe and pyspark>=4.0.0.
### Setup process
### Screenshots
## Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
## Related PRs
## Docs link
https://bito.ai#summarystart|
### Summary by Bito
This pull request adds support for the new
pyspark.sql.classic.dataframe.DataFrame
type for compatibility with
pyspark
version 4.0.0 and above. It includes new classes for reading and writing this DataFrame type, along with serialization and deserialization handlers. Plugin requirements have also been updated accordingly.
flyteorg/flytekit