#4056 [Core feature] Create or document a fast way to query the status of large workflows Issue created by Tom-Newton Motivation: Why do you think this is important? For large workflows with ~1000s of nodes its difficult to monitor the overall progress. At this scale we need to be able to get things like counts of how many nodes or sub-workflows are in each status. e.g. 900 success, 200: unknown 2 failed, 98: in progress. I think supporting large workflows like this would be a valuable feature and its critical to what I want to do with flyte. Goal: What should the final outcome look like, ideally? I think ideal would be an interface similar to

remote.sync_execution(sync_nodes=True)

but fast for large workflows. This would be very flexible. Other interfaces would also be fine - I'm mostly just interested in it being fast. Describe alternatives you've considered The flyte UI: It displays lists and graphs but at the scale of 1000s of nodes these are impossible to parse by eye. Additionally in tends to crash my browser.

flytectl get execution

Can get information about nodes when using

--details

but it seems to be incomplete. Writing to a

.yaml

file and searching, I find quite a lot of nodes are missing.

flytekit

remote.sync_execution(sync_nodes=True)

This does fetch all the important information and could certainly be parsed by some python code to extra whatever metrics are needed. The problem is that it takes about 12 minutes to run on a workflow with 3000 nodes. EDIT: It actually doesn't fetch information about nodes that haven't started processing yet. So nodes that would show with unknown status on the UI are missed. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? • Yes Have you read the Code of Conduct? • Yes flyteorg/flyte