https://flyte.org
Join Slack
Is there support in flyte to trigger workflows via sqs (aws) or pubsub (gcp)? Or is the only way to ...
# flyte-support
h
Is there support in flyte to trigger workflows via sqs (aws) or pubsub (gcp)? Or is the only way to achieve this to create a custom plugin for flyte? Or a custom service for either aws/gcp that triggers the workflow via sdk?
👀 1
could this be done as a custom Sensor?
g
yes, you can create a custom sensor that listen to sqs
h
is this a common pattern to trigger workflows via events? Trying to understand better typical workflow execution life cycles. Seems like UI based triggers and scheduled triggers are covered in the docs. My use case is triggering workflows via files uploaded to s3 buckets.
g
> is this a common pattern to trigger workflows via events yes, a lot of people do that, so that’s why we add sensor to flyte
> My use case is triggering workflows via files uploaded to s3 buckets. you can try our file sensor
h
yeah i looked, it seems like that is for 1 specific file only
g
you want to watch a directory?
h
i would need to trigger workflows for any file under a prefix in s3
yeah
g
we were talking about this internally. would you mind helping us create an issue here. [flyte-core]
u
h
awesome i will do that
g
add your use cases in the description. will prioritize this. thank you
h
g
thank you so much
@hallowed-dog-74273 there is an alternative by using aws lambda. check out this https://flyte.org/blog/build-an-event-driven-neural-style-transfer-application-using-aws-lambda
e
How does the current sensor implementation work? Does the underlying workflow need to run on a schedule or will the presence of a sensor in the workflow mean that it's automatically triggered / runs on a separate control loop?
Ah, I guess it's an agent that runs separately?
g
Ah, I guess it’s an agent that runs separately?
yes. but the sensor is also a task in the workflow
e
Is there any place in the docs that clearly outlines the lifecycle of how a sensor operates? I guess I'm a little unclear where the task execution comes into play/if a workflow that uses a sensor to trigger execution needs to be scheduled, or there's a separate control loop that runs the sensor logic Use case I'm thinking about is triggering a workflow upon successful completion of a prior workflow and using a sensor for that
👍 1
g
there’s a separate control loop that runs the sensor logic
tldr, we run the sensor logic in the agent deployment instead of creating a new pod to do that
👍 1
e
Amazing, thanks - that answers my main question
TLDR is we're looking to move to a common surface area for triggering across an argo-based system as well as Flyte, and being able to implement an arbitrary sensor (ie. that listens to an SQS queue) will be a very helpful layer of indirection
g
the problem of the current sensor is that it can only run once. for example, if you use file sensor and upload data to a specific bucket, sensor will complete and propeller will run downstream task. To watch the bucket again, you need to launch another workflow. We plan to add triggers that runs in flyteadmin probably that watch the event until you delete it. and it’s not a task. it’s completely new thing in flyteadmin
yes, I think trigger is what you want. should have a while loop somewhere in admin to watch the certain event.
e
Yes this sounds like what we’d need. Ideally we could package a container or run a decorated function for the trigger and define arbitrary logic
m
We also need this feature. In our case we receive data daily and currently we have scheduled Flyte workflow to execute every 4 hours (because we don't really have control over when data will arrive) . If we get above mentioned mechanism in which Flyte will re-run the workflow next day when data arrives then we don't have to schedule the workflow every 4 hours.
g
cc @high-park-82026 we discussed this yesterday.
e
Rather than running this in flyteadmin, what do you think about another controller that manages a (user-defined) triggers - and (ie 
@trigger
-decorated function), and expose an API that determines the downstream action based on the return value? Also happy to help make this a reality, this would be a huge help. Is the work planned?
35 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by