@ancient-wolf-19325 I need a sensor that automatically triggers based on items on a queue

Flyte supports sensor tasks that can wait for external events, but there is no built-in, production-ready sensor for generic queue-based triggers (like SQS, Pub/Sub, or Kafka) in the core Flytekit as of now. However, you can implement a custom sensor plugin in Flytekit that polls a queue and completes when a condition is met. The user container task plugin example shows how to build a sensor that waits for a file; you can adapt this pattern to poll a queue instead. Additionally, there is ongoing discussion and feature requests for native Pub/Sub or queue-based sensors, but these are not yet part of the standard Flyte release (see issue). For now, you must implement your own sensor logic or use an external service to trigger Flyte workflows programmatically when a queue event occurs. Would you like a step-by-step example of how to implement a custom queue sensor in Flyte? Sources: - User container task plugins: sensor example - Feature request: Event Based Sensors for bucket upload events (Pub/Sub) 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 is it possible for a workflow to be triggered automatically by sensor, and as well as manually triggered by user

Yes, a Flyte workflow can be triggered both automatically by a sensor (such as a file sensor or a custom event-based sensor) and manually by a user. Sensors can be implemented as tasks within a workflow that wait for external events, and users can also launch workflows directly via the UI, CLI, or SDK (user container task plugins: sensor example, running workflows). This dual-trigger pattern is supported: the sensor task can be part of the workflow DAG, and users can manually launch the workflow at any time, bypassing or including the sensor logic as needed. There is no restriction in Flyte that limits a workflow to only one type of trigger. Would you like more detail or a code example? Sources: - user container task plugins: sensor example - running workflows 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 can you help m write a custom sensor which reads from messaging queue

Yes, you can write a custom sensor in Flytekit by subclassing

BaseSensor

and implementing the

poke

method to check your messaging queue. The sensor can be used as a task in your workflow. Here’s a minimal example for a custom sensor that polls a queue (you’ll need to fill in the actual queue logic):

from flytekit.sensor.base_sensor import BaseSensor

class QueueSensor(BaseSensor):
    def __init__(self, name: str, config=None, **kwargs):
        super().__init__(name=name, sensor_config=config, **kwargs)

    async def poke(self, queue_name: str) -> bool:
        # Replace this with your queue polling logic
        # For example, check if there are messages in the queue
        # Return True if a message is found, False otherwise
        return check_queue_for_message(queue_name)

You can then use this sensor in your workflow as shown in the sensor plugin example and file sensor example. Would you like a more detailed, step-by-step guide or a full workflow example? Sources: - user container task plugins: sensor example - Flyte 1.10: custom sensor example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 How would I deploy this sensor to my currently deployed flyte instance?

To deploy your custom sensor to an existing Flyte instance, you need to build a Docker image containing your sensor code, push it to your registry, and update your workflow to use this image. Then, register your workflow and tasks with Flyte. This is the same process as deploying any Flyte workflow with custom code (user container task plugins: sensor example, Flyte deployment lifecycle). If your sensor is implemented as a Flyte agent (backend plugin), you must also build and deploy the agent service, update the FlytePropeller configuration to route the relevant task type to your agent, and restart FlytePropeller (developing agents, deploying agents to the Flyte sandbox). Would you like a step-by-step deployment guide for either approach? Sources: - developing agents - deploying agents to the Flyte sandbox - user container task plugins: sensor example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.