#6552 [Core Feature][Not ready] Allow to configure agent status update frequency
Pull request opened by
popojk
## Tracking issue
Related to
#5733
## Why are the changes needed?
In the current Flyte backend, a single connector plugin is registered to handle all connector task types. The configuration file allows setting a global cache resync interval and rate limit for the connector plugin. However, these settings apply uniformly across all connector task types, limiting flexibility.
To improve configurability, we propose initializing a separate connector plugin instance for each configured connector task type. This approach enables fine-grained control, allowing distinct cache resync intervals and rate limits per task type.
The proposed system architecture is illustrated below:
[截圖 2025-07-30 下午1 28 02](https://private-user-images.githubusercontent.com/66259759/472313929-6640fcdc-12bd-4639-8b20-58015e14611b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM4NTUxNDUsIm5iZiI6MTc1Mzg1NDg0NSwicGF0aCI6Ii82NjI1OTc1OS80NzIzMTM5MjktNjY0MGZjZGMtMTJiZC00NjM5LThiMjAtNTgwMTVlMTQ2MTFiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzMwVDA1NTQwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc4MWE4NGMwZGQ2Nzg3MDFlMDJkZWM2NGI1MzljNTY1ZWYyZDg2MGU3NDMwODQzOTFkZTI0ODgxMDVhMDUzZTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.AKfn91peRZRSMDRCzeK3HyclN9QDkEfWApa9jLqwf14)
In Flyte config file, we may configure cache resync interval and rate limit for each connector deployment
[截圖 2025-07-30 下午1 32 09](https://private-user-images.githubusercontent.com/66259759/472315347-51aef483-086a-4228-8e46-ce52e79e746f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM4NTUxNDUsIm5iZiI6MTc1Mzg1NDg0NSwicGF0aCI6Ii82NjI1OTc1OS80NzIzMTUzNDctNTFhZWY0ODMtMDg2YS00MjI4LThlNDYtY2U1MmU3OWU3NDZmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzMwVDA1NTQwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAxNjQ5YTQwMzRiZGFiNWM1OWM1ZGQzNGYzZDEwZDg2ZjljOTMxZDBkYzVkZTQwODg0ZjczNjFlZDkyMWM1NjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.DeYMuqxUWw-UFf9LZObS7HVWRnKtuUCwMpROU2nhsxQ)
## What changes were proposed in this pull request?
1. In Propeller TaskHandler, add a watcher to watch message from connector global watcher to update/create defaultPlugins in TaskHandler.
2. In PluginMachinery registry, add a
pluginChan
channel for connector global watcher to send message.
3. In webapi connector, add global watcher. The global watcher will watch all deployments defined in config file on routine basis to retrieve supported task types from each deployment. A connector plugin will be registered for a task type of each deployment.
## How was this patch tested?
### Labels
Please add one or more of the following labels to categorize your PR:
•
added: For new features.
This is important to improve the readability of release notes.
### Setup process
### Screenshots
## Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
## Related PRs
## Docs link
https://bito.ai#summarystart|
### Summary by Bito
This pull request significantly enhances the Flyte backend's connector plugin system, allowing for individual configuration of cache resync intervals and rate limits for each task type. It introduces a global watcher for dynamic management of connector deployments and refines the plugin registration process, improving overall flexibility and performance.
flyteorg/flyte