https://flyte.org logo
#ask-the-community
Title
# ask-the-community
c

Chandrashekhar Kotekar

02/22/2024, 9:54 AM
Is it possible to stop the
sensor
to look for arrival of some file after certain time? For example - give up waiting for the data after 2 hours / days / weeks.
s

Samhita Alla

02/23/2024, 6:41 AM
i don't think we support this today, but it should be possible to add this functionality to
FileSensor
. is this something you're willing to contribute? cc @Kevin Su
c

Chandrashekhar Kotekar

02/23/2024, 7:31 AM
Thank you
k

Kevin Su

02/23/2024, 7:42 AM
yes, it’s doable. you can add that to sensor config
c

Chandrashekhar Kotekar

02/23/2024, 7:49 AM
Hey @Kevin Su thanks for quick reply. Is this functionality already available or will it be added in future?
k

Kevin Su

02/23/2024, 7:54 AM
not available now. just curious, what’s the use case for this? are you able to launch the workflow after certain time?
c

Chandrashekhar Kotekar

02/23/2024, 8:37 AM
We have created a launch plan which runs workflow every 2 hours. We wanted to try using sensor to begin executing tasks when files arrive in S3 bucket. So scenario is: Suppose first execution (starting at 9 AM) sensor is looking for the data to arrive which won't arrive before 12 PM (time at which data arrives varies) but since we have scheduled launchplan to run workflow every 2 hours, next execution will begin at 11 AM and both executions will keep looking for same data. This can lead to double processing of the data. So idea was like if we can give timeout to the file sensor like - stop looking for data arrival after 1 hour then we can avoid this problem of double processing. Please let me know if there is any other
Flyte Specific
way to handle such scenario.
Can we just extend the
FileSensor
and add this functionality? Or do we need to register this custom sensor before using it?
k

Ketan (kumare3)

02/25/2024, 11:01 PM
Sensor should have timeouts - absolutely
@Kevin Su / @Samhita Alla why no timeouts we should allow them Like regular tasks
s

Samhita Alla

02/26/2024, 5:35 AM
absolutely. would you mind creating an issue, @Chandrashekhar Kotekar? if you could contribute, that'd be great!
c

Chandrashekhar Kotekar

02/26/2024, 8:10 AM
@Samhita Alla will create an issue for this and
maybe
I will contribute it.