Is there a way to save state for an agent task? I ...
# flyte-connectorsc
colossal-nightfall-74781
03/18/2025, 2:17 PMIs there a way to save state for an agent task? I believe the ResourceMetadata are immutable and won't change after calling the agent's create method.
We currently create a custom metastore to save attributes for individual tasks and update them on each invocation. It would be great if we could alter the resource metadata instead.
g
glamorous-carpet-83516
03/18/2025, 5:29 PMwhat kind of attributes do you want to save? would to understand more about the use case
c
colossal-nightfall-74781
03/18/2025, 5:51 PMIn one example we do a file count and wait for a certain time until the file count becomes stable. We don't know how many files there will be and when they will be delivered only the folder and the approximate time span it takes to deliver files. We need to store the file count and the time of last sync in our metastore
g
glamorous-carpet-83516
03/18/2025, 5:59 PMso it’s kind of sensor, the task succeed until the file count becomes stable
glamorous-carpet-83516
03/18/2025, 6:04 PMbut could you do something like this
Copy code
async def get(self, resource_meta: JobMetadata, **kwargs) -> Resource:
file_count = get_file_count()
if file_count > 100 and current_time - resource_meta.start_time > 10:
return Resource(TaskExecution.SUCCEEDED)
return Resource(TaskExecution.RUNNING)c
colossal-nightfall-74781
03/18/2025, 7:42 PMno, I don't know how many files there will be (sometimes there won't be any, sometimes many) and when the files will be delivered (current timeout limit is 4 days)
4 Views