Hi, I am trying to write a

sensor

which is supposed to check if an object exists in an

S3 bucket

or not. I am referring to this tutorial but I would like to know how to package this sensor + workflow together? Do we keep in a docker container? or register sensor in Flyte?

@millions-night-34157 you should look at Flyte agent framework - which is soft released and is being released in 1.9 that also contains s3/file sensor and auto support for any airflow sensor, and runs as a long running service

Are you referring to this page? https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/extend_flyte/agent_service.html#what-is-an-agent

And a great video talking about agent. The FileSensoerAgent example is at 11:59.

cc @glamorous-carpet-83516 share the example

Hi @freezing-airport-6809 We are using

flytekit==1.8.1

. Did you meant to say that flytekit 1.9 will come with this s3/ file sensor?

Hi @millions-night-34157 • For the sensor, we are adding another class on top of agent, which makes adding a new agent much easier. check out this pr • if you want to submit the job to the external service, such as BigQuery. you can refer to this agent The tasks run on agents, so if you create a new task, you need to create an agent as well. in the sensor pr, I create a sensor task and agent. The task inheriting

AsyncAgentExecutorMixin

can be run locally. In the

execute

, we actually call agent create / get to run the task. if you don’t want to run on agent, you can override the

execute

by yourself.

Thanks @glamorous-carpet-83516 for the examples. I am reading PR now.

Here is an airflow agent, allows you to use airflow operator / sensor in the flyte workflow. every airflow tasks are running on this agent. Really appreciate any feedback!

my use case is like: we have one pipeline which usess input data from S3 objects. We want to write sensor OR agent OR something else which will allow the pipeline to keep waiting until S3 object has become available . Once S3 object is available, pipeline should start executing the actual tasks.

ok, seems like file sensor PR can be used in your use cases. you can pull the PR, and give it a shot.

that's great. Thanks 🙂

You don’t have to modify it

yes, we’ll merge it soon

that's cool. If I've understood it correctly - this file sensor will be available from flytekit version 1.9 . Am I right?

correct

Perfect. Do we know when v 1.9 will be available? 🙂

next week. will let you know once 1.9 is out

Awesome. Looking forward to it