https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Alex Beach

01/04/2024, 10:14 PM
Is there support in flyte to trigger workflows via sqs (aws) or pubsub (gcp)? Or is the only way to achieve this to create a custom plugin for flyte? Or a custom service for either aws/gcp that triggers the workflow via sdk?
could this be done as a custom Sensor?
k

Kevin Su

01/04/2024, 10:45 PM
yes, you can create a custom sensor that listen to sqs
a

Alex Beach

01/04/2024, 10:50 PM
is this a common pattern to trigger workflows via events? Trying to understand better typical workflow execution life cycles. Seems like UI based triggers and scheduled triggers are covered in the docs. My use case is triggering workflows via files uploaded to s3 buckets.
k

Kevin Su

01/04/2024, 10:53 PM
> is this a common pattern to trigger workflows via events yes, a lot of people do that, so that’s why we add sensor to flyte
> My use case is triggering workflows via files uploaded to s3 buckets. you can try our file sensor
a

Alex Beach

01/04/2024, 10:54 PM
yeah i looked, it seems like that is for 1 specific file only
k

Kevin Su

01/04/2024, 10:56 PM
you want to watch a directory?
a

Alex Beach

01/04/2024, 10:56 PM
i would need to trigger workflows for any file under a prefix in s3
yeah
k

Kevin Su

01/04/2024, 10:58 PM
we were talking about this internally. would you mind helping us create an issue here. [flyte-core]
a

Alex Beach

01/04/2024, 10:58 PM
awesome i will do that
k

Kevin Su

01/04/2024, 10:59 PM
add your use cases in the description. will prioritize this. thank you
a

Alex Beach

01/04/2024, 11:32 PM
k

Kevin Su

01/04/2024, 11:39 PM
thank you so much
@Alex Beach there is an alternative by using aws lambda. check out this https://flyte.org/blog/build-an-event-driven-neural-style-transfer-application-using-aws-lambda
r

Rahul Mehta

01/08/2024, 1:31 PM
How does the current sensor implementation work? Does the underlying workflow need to run on a schedule or will the presence of a sensor in the workflow mean that it's automatically triggered / runs on a separate control loop?
Ah, I guess it's an agent that runs separately?
k

Kevin Su

01/08/2024, 7:59 PM
Ah, I guess it’s an agent that runs separately?
yes. but the sensor is also a task in the workflow
r

Rahul Mehta

01/08/2024, 9:22 PM
Is there any place in the docs that clearly outlines the lifecycle of how a sensor operates? I guess I'm a little unclear where the task execution comes into play/if a workflow that uses a sensor to trigger execution needs to be scheduled, or there's a separate control loop that runs the sensor logic Use case I'm thinking about is triggering a workflow upon successful completion of a prior workflow and using a sensor for that
k

Kevin Su

01/08/2024, 9:32 PM
there’s a separate control loop that runs the sensor logic
tldr, we run the sensor logic in the agent deployment instead of creating a new pod to do that
r

Rahul Mehta

01/08/2024, 9:33 PM
Amazing, thanks - that answers my main question
TLDR is we're looking to move to a common surface area for triggering across an argo-based system as well as Flyte, and being able to implement an arbitrary sensor (ie. that listens to an SQS queue) will be a very helpful layer of indirection
k

Kevin Su

01/08/2024, 9:36 PM
the problem of the current sensor is that it can only run once. for example, if you use file sensor and upload data to a specific bucket, sensor will complete and propeller will run downstream task. To watch the bucket again, you need to launch another workflow. We plan to add triggers that runs in flyteadmin probably that watch the event until you delete it. and it’s not a task. it’s completely new thing in flyteadmin
yes, I think trigger is what you want. should have a while loop somewhere in admin to watch the certain event.
r

Rahul Mehta

01/09/2024, 1:47 PM
Yes this sounds like what we’d need. Ideally we could package a container or run a decorated function for the trigger and define arbitrary logic
c

Chandrashekhar Kotekar

01/26/2024, 12:38 PM
We also need this feature. In our case we receive data daily and currently we have scheduled Flyte workflow to execute every 4 hours (because we don't really have control over when data will arrive) . If we get above mentioned mechanism in which Flyte will re-run the workflow next day when data arrives then we don't have to schedule the workflow every 4 hours.
k

Kevin Su

01/26/2024, 6:45 PM
cc @Haytham Abuelfutuh we discussed this yesterday.
r

Rahul Mehta

02/01/2024, 2:58 PM
Rather than running this in flyteadmin, what do you think about another controller that manages a (user-defined) triggers - and (ie
@trigger
-decorated function), and expose an API that determines the downstream action based on the return value? Also happy to help make this a reality, this would be a huge help. Is the work planned?