Is there support in flyte to trigger workflows via sqs (aws) or pubsub (gcp)? Or is the only way to achieve this to create a custom plugin for flyte? Or a custom service for either aws/gcp that triggers the workflow via sdk?

yes, you can create a custom sensor that listen to sqs

is this a common pattern to trigger workflows via events? Trying to understand better typical workflow execution life cycles. Seems like UI based triggers and scheduled triggers are covered in the docs. My use case is triggering workflows via files uploaded to s3 buckets.

> is this a common pattern to trigger workflows via events yes, a lot of people do that, so that’s why we add sensor to flyte

yeah i looked, it seems like that is for 1 specific file only

you want to watch a directory?

i would need to trigger workflows for any file under a prefix in s3

we were talking about this internally. would you mind helping us create an issue here. [flyte-core]

⭐ Create a new Flyte Core Feature issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+

awesome i will do that

add your use cases in the description. will prioritize this. thank you

thank you so much

How does the current sensor implementation work? Does the underlying workflow need to run on a schedule or will the presence of a sensor in the workflow mean that it's automatically triggered / runs on a separate control loop?

Ah, I guess it’s an agent that runs separately?

yes. but the sensor is also a task in the workflow

Is there any place in the docs that clearly outlines the lifecycle of how a sensor operates? I guess I'm a little unclear where the task execution comes into play/if a workflow that uses a sensor to trigger execution needs to be scheduled, or there's a separate control loop that runs the sensor logic Use case I'm thinking about is triggering a workflow upon successful completion of a prior workflow and using a sensor for that

there’s a separate control loop that runs the sensor logic

tldr, we run the sensor logic in the agent deployment instead of creating a new pod to do that

Amazing, thanks - that answers my main question

the problem of the current sensor is that it can only run once. for example, if you use file sensor and upload data to a specific bucket, sensor will complete and propeller will run downstream task. To watch the bucket again, you need to launch another workflow. We plan to add triggers that runs in flyteadmin probably that watch the event until you delete it. and it’s not a task. it’s completely new thing in flyteadmin

Yes this sounds like what we’d need. Ideally we could package a container or run a decorated function for the trigger and define arbitrary logic

We also need this feature. In our case we receive data daily and currently we have scheduled Flyte workflow to execute every 4 hours (because we don't really have control over when data will arrive) . If we get above mentioned mechanism in which Flyte will re-run the workflow next day when data arrives then we don't have to schedule the workflow every 4 hours.

cc @Haytham Abuelfutuh we discussed this yesterday.

Rather than running this in flyteadmin, what do you think about another controller that manages a (user-defined) triggers - and (ie

@trigger

-decorated function), and expose an API that determines the downstream action based on the return value? Also happy to help make this a reality, this would be a huge help. Is the work planned?