hi everyone 👋 new to flyte and just learned that sensing à la airflow is considered an anti-pattern (see screenshot). i was wondering how to go about this the right way in flyte as an example, say a workflow has a task (at the beginning or further downstream) that relies on an S3 key to materialise as input for processing. in airflow, i'd sense the bucket using S3KeySensor. with flyte, would this process be implemented outside the workflow? for example using a lambda function that gets triggered by the key and starts the flyte pipeline? and would i usually break up my flyte pipeline into multiple pipelines if this step occurred somewhere downstream rather than right at the start?

That being said we are adding support for it - you can build one but @Kevin Su has support for all airflow sensors

@Alexander Walther check out airflow agent, which allows to use airflow sensor and operator in the flyte workflow. https://github.com/flyteorg/flytekit/pull/1725

@Kevin Su thanks for the PR! @Ketan (kumare3) thanks 🙂 is this PR going to invalidate the bullet on sensing being an anti-pattern, or will this still be somewhat true post merge for flyte?

Haha - I think we should still educate folks the problems with sensing