#823 Add pandera plugin Pull request opened by cosmicBboy This pull request introduces the new

flyteplugins-pandera

plugin, which adds Pandera schema validation support for pandas, Polars, and PySpark SQL DataFrame types in Flyte. It includes plugin implementation, registration, configuration, and comprehensive usage examples for each supported backend. The plugin ensures DataFrames are validated with Pandera schemas at task boundaries and generates HTML validation reports, with careful handling to avoid redundant reporting in nested task scenarios. The most important changes are: Plugin Implementation and Registration: • Added the new

flyteplugins-pandera

plugin, including core modules for type transformers, validation configuration, and HTML report rendering for Pandera-typed DataFrames (

pandas

,

polars

,

pyspark_sql

). [1]], [2]], [3]], [4]], [5]], [6]], [7]]) • Registered the plugin as a Flyte type transformer entry point, enabling automatic integration when installed. ([plugins/pandera/pyproject.tomlR1-R97]) Documentation and Examples: • Added detailed documentation in

README.md

files for both the plugin and usage examples, explaining installation, supported backends, runtime behavior, and troubleshooting tips. [1]], [2]]) • Provided comprehensive example scripts demonstrating Pandera validation with pandas, Polars, and PySpark SQL DataFrames, including error handling modes and Flyte task integration. [1]], [2]], [3]]) Build and Release Integration: • Updated the GitHub Actions workflow to include the new

plugins/pandera

directory in the publish matrix, ensuring the plugin is built and released with other plugins. ([.github/workflows/publish.ymlR91]) These changes collectively enable robust, schema-driven DataFrame validation in Flyte workflows using Pandera, with first-class support for pandas, Polars, and PySpark SQL backends. flyteorg/flyte-sdk