#6478 [Core feature] Add support pyspark.sql.classic.dataframe.DataFrame transformer Issue created by pingsutw Motivation: Why do you think this is important?

pyspark.sql.classic.dataframe.DataFrame

is a new type of dataframe; it's not recognized by the current structured dataset encoder/decoder. Therefore, the type transformer will fail to serialize and deserialize it. https://github.com/flyteorg/flytekit/actions/runs/15334631361/job/43149503509 Goal: What should the final outcome look like, ideally? Add support

pyspark.sql.classic.dataframe.DataFrame

structured dataset encoder/decoder Describe alternatives you've considered NA Propose: Link/Inline OR Additional context NA Are you sure this issue hasn't been raised already? • Yes Have you read the Code of Conduct? • Yes flyteorg/flyte