Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello! I've been building <https://provero.org/|Provero>, an open-source, vendor-neutral data quality engine. Think declarative quality checks defined in YAML or code, executed via SQL against any backend (DuckDB, Postgres, Snowflake, BigQuery, etc).

It just got native Flyte support. You can validate DataFrames at task boundaries using Annotated type hints, same pattern as Pandera:

ValidatedDF = Annotated[pd.DataFrame, ProveroSuite(
      checks=[CheckConfig(check_type="not_null", column="id")],
)]

@task
  def load_data() -&gt; ValidatedDF:
      return pd.read_parquet("data.parquet")

Quality reports show up as Flyte Decks in the console. Also works as a standalone task or decorator if you prefer YAML-driven checks.

Repo: <https://github.com/provero-org/provero>

Still early stage, would love feedback from anyone doing data quality in Flyte pipelines.

looks awfully similar to pandera :upside_down_face:

Fair point, the Flyte integration pattern with Annotated type hints is definitely inspired by Pandera.

The core difference is that Provero pushes checks down as SQL to the source (Postgres, Snowflake, BigQuery) instead of validating in-memory DataFrames. So for large tables you skip the full data pull. It also bundles anomaly detection, alerting, and data contracts into the same YAML config.

The Flyte type checker integration just reuses the pattern you established because it works really well. :relaxed:

&gt; Provero pushes checks down as SQL to the source (Postgres, Snowflake, BigQuery) instead of validating in-memory DataFrames
yep! pandera does the same with the `ibis` integration (and soon with the `narwhals` ) integration

&gt; It also bundles anomaly detection, alerting, and data contracts into the same YAML config
this is a great idea! will see about adding this to pandera too

Didn't know about the Ibis integration, that's great. The YAML-first approach and built-in anomaly detection + contracts are where Provero focuses most. Would be cool to see that in Pandera too.