#6539 [Plugin]Build a New Plugin for Apache Beam on Flyte Issue created by vaibhav-research on <!date^1753222874^{date_short}|2025-07-22T22:21:14Z> ### Plugin Proposal: Apache Beam Integration with Flyte #### Overview This proposal aims to build a new Flyte plugin that enables users to run Apache Beam pipelines as native Flyte tasks. Apache Beam provides a unified programming model for batch and streaming data processing, supporting multiple runners like Google Dataflow, Apache Flink, Spark, and DirectRunner. The plugin will allow users to author Beam jobs using the Flyte SDK and submit them to a configured Beam runner from within a Flyte workflow. --- #### Motivation • Apache Beam is a widely adopted industry and research standard for portable, scalable data pipelines. • Currently, Flyte supports Spark, Dask, and other distributed engines, but lacks support for unified batch/streaming pipelines like Beam. • This plugin will enable flexible, scalable ML/data engineering workflows by combining Beam's runner-agnostic pipeline design with Flyte's orchestration power. • This aligns well with Flyte's mission to be runner-agnostic and cloud-native. --- #### Proposed Features • A new

BeamTask

in FlyteKit to define Beam jobs as tasks. • Support for: • Runners:

DirectRunner

,

DataflowRunner

, and optionally

FlinkRunner

• Input/output file handling via Flyte’s native types • Pipeline parameterization through Flyte inputs • Plugin type:

flytekit-only

initially (no backend changes needed unless advanced features are later added) --- #### Deliverables •

flytekitplugins-beam

Python package • Documentation with usage examples and integration tests • Optional: Blog post or tutorial showcasing Beam+Flyte integration --- #### References • Apache Beam Python SDK • DataflowRunner • Beam Quickstart --- #### Contribution Intent I'd like to contribute this plugin and collaborate with Flyte maintainers to finalize the design and implementation. Please let me know if there’s an existing effort in progress or if there are constraints I should be aware of before starting development. --- #### Goal To create a high-impact plugin that expands Flyte's data processing capabilities and adds value to the ML and data engineering community. --- /cc @flyteorg/maintainers flyteorg/flyte