hi guys, I have managed to create my custom agent/sensor/task leveraging flyte agent framework. It simplifies a lot of complicated task and works fine. now I am decoupling my custom agent/task from the flytekit SDK and deployed it standalone to be managed by my org/teams. And then I am seeing an error that I have never seen before and did not have much info to go deeper per se. from in propeller i only see the permission error:

s":"kk-flyte-dev3","res_ver":"10264174790","routine":"worker-2","src":"executor.go:297","wf":"shuliang-projects:development:gnn_agent_sensor_mpijob_ei_faro.distributed_multi_node_graph_training"},"level":"debug","msg":"Transitioning/Recording event for workflow state transition [Failing] -\u003e [Failed]","ts":"2024-09-23T07:20:30Z"}

any idea what’s happening? is there some other place we can look at for what caused permission of creating the custom task?

I've never seen this error before

I have not tried the local execution though. still figuring out where is this even coming from.. e.g. from propeller? admin, etc..

Transitioning/Recording event

I think it's from propeller

yeah I saw the log from propeller, but didnt have more information why propeller failed like that…

this is a hard one, let me ask some OSS contributors.

sure. what do you need? just you know that the task execution is readable when not running stanalone?

I haven't seen this one pop up at all in our agent (and we've been running with agents for ~ 6 months now)

I mean print the

TaskExecution.Phase

to check if it is readable for propeller or not

Thanks for the follow up Han-Ru, Ethan and Kevin. I think I might know what went wrong, will update when I had more data later today.

nice, thank you so much