hi everyone 👋
new to flyte and just learned that sensing à la airflow is considered an anti-pattern (see screenshot). i was wondering how to go about this the right way in flyte
as an example, say a workflow has a task (at the beginning or further downstream) that relies on an S3 key to materialise as input for processing. in airflow, i'd sense the bucket using S3KeySensor. with flyte, would this process be implemented outside the workflow? for example using a lambda function that gets triggered by the key and starts the flyte pipeline? and would i usually break up my flyte pipeline into multiple pipelines if this step occurred somewhere downstream rather than right at the start?
08/31/2023, 6:11 PM
That being said we are adding support for it - you can build one but @Kevin Su has support for all airflow sensors