Is it possible to stop the

sensor

to look for arrival of some file after certain time? For example - give up waiting for the data after 2 hours / days / weeks.

i don't think we support this today, but it should be possible to add this functionality to

FileSensor

. is this something you're willing to contribute? cc @Kevin Su

Thank you

yes, it’s doable. you can add that to sensor config

Hey @Kevin Su thanks for quick reply. Is this functionality already available or will it be added in future?

not available now. just curious, what’s the use case for this? are you able to launch the workflow after certain time?

We have created a launch plan which runs workflow every 2 hours. We wanted to try using sensor to begin executing tasks when files arrive in S3 bucket. So scenario is: Suppose first execution (starting at 9 AM) sensor is looking for the data to arrive which won't arrive before 12 PM (time at which data arrives varies) but since we have scheduled launchplan to run workflow every 2 hours, next execution will begin at 11 AM and both executions will keep looking for same data. This can lead to double processing of the data. So idea was like if we can give timeout to the file sensor like - stop looking for data arrival after 1 hour then we can avoid this problem of double processing. Please let me know if there is any other

Flyte Specific

way to handle such scenario.

Sensor should have timeouts - absolutely

absolutely. would you mind creating an issue, @Chandrashekhar Kotekar? if you could contribute, that'd be great!

@Samhita Alla will create an issue for this and

maybe

I will contribute it.