@ancient-wolf-19325 We have a file sensor which waits for a file to arrive in S3 bucket. If this file arrives after couple of hours, file sensor does not return and keeps waiting for the file even though now file is available. What could be the issue for this?

The issue might be related to the lack of a timeout configuration for the file sensor, causing it to wait indefinitely even if the file arrives after a delay. According to Flyte GitHub issue #5035, there is currently a need for a timeout feature in file sensors to prevent them from waiting indefinitely when the expected file does not arrive within a specified time frame. Would you like more detailed information on how to configure a timeout for your file sensor or other potential causes and solutions for this issue? Sources: - Flyte GitHub issue #5035 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 if timeout is not set and file arrives late then also file sensor should be able to detect it.

If the file sensor should detect the file even if it arrives late and it is not doing so, the issue might be due to the sensor's polling mechanism or a misconfiguration in the sensor setup. According to the FileSensor example, the sensor should continuously poll for the file's existence at regular intervals. Ensure the following: 1. Polling Interval: Verify that the polling interval is correctly set and the sensor is actively polling for the file. 2. Correct Path: Confirm that the path specified in the sensor matches the actual file path in the S3 bucket. 3. Permissions: Ensure that the sensor has the necessary permissions to access the S3 bucket and detect the file. Would you like more details on configuring the polling interval or other aspects of the file sensor setup? Sources: - FileSensor example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 if we restart workflow execution then during next execution file sensor detects presence of file. So polling interval and permissions are not an issue. Does file sensor increases time between two "polls" after certain interval?

Flyte's file sensor does not inherently increase the time between polls. It should consistently poll at the interval specified during its configuration. If the file sensor detects the file upon restarting the workflow but not during continuous execution, it might indicate an issue with the sensor's state management or a potential bug. According to the Flyte discussion, there is no mention of the sensor increasing the polling interval over time. The sensor should maintain a consistent polling interval as configured. Would you like more details on how to configure a consistent polling interval or investigate other potential issues with the sensor's state management? Sources: - Flyte discussion - FileSensor example 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.