Channels
#ask-the-community
Title
# ask-the-community
l
Len Strnad
07/11/2023, 6:09 PMHey, all! I am looking for a way to get an execution’s tasks and corresponding durations using an existing python API. Is there a way to accomplish this?
Is there anything in flytekit that might help me here that I am missing?
This is what I have so far:
Copy code
from flytekit.remote import FlyteRemote, FlyteWorkflowExecution, FlyteTaskExecution
from flytekit.configuration import Config
remote = FlyteRemote(
config=Config.for_endpoint(endpoint=endpoint),
default_project=project,
default_domain=domain,
)
# specific execution
execution = remote.fetch_execution(name="ffc0e12bb3e314fae9aa")
ex = remote.sync_execution(execution=execution, sync_nodes=True)
# ['n0'] is the node I am interested in
node_times = {x.metadata.spec_node_id: x.closure.duration.total_seconds() / 60 \
for x in ex.node_executions['n0'].executions}Gives me something like
Copy code
{'end-node': 0.0,
'n0': 2.698262133333333,
'n1': 2.202614066666667,
'n2': 1.59931375,
'n3': 1.8271749,
'n4': 30.83562241666667,
'n5': 0.2830597833333333,
'n6': 0.3337019666666666,
'start-node': 0.0}j
Jay Ganbat
07/11/2023, 8:24 PMi think its sequential on how its defined in the dynamic tasks, are there no node metadata associated with node executions?
l
Len Strnad
07/11/2023, 8:25 PM Copy code
example = ex.node_executions['n0'].executions[2]
example.metadata Copy code
retry_group: "0"
spec_node_id: "end-node"and
Copy code
example.id Copy code
node_id: "n0-0-n1"
execution_id {
project: "..."
domain: "..."
name: "ffc0e12bb3e314fae9aa"
}j
Jay Ganbat
07/11/2023, 8:29 PMhow about
task_node_metadatal
Len Strnad
07/11/2023, 8:30 PMas an attribute of
examplej
Jay Ganbat
07/11/2023, 8:31 PMso i can get the task info from this
Copy code
execution.node_executions.get("n1").closure.task_node_metadatal
Len Strnad
07/11/2023, 8:35 PMexecution.node_executions.get('n0').closure.task_node_metadata🤷
I am on `'1.4.2'`of flytekit, what about you @Jay Ganbat?
j
Jay Ganbat
07/11/2023, 8:44 PMsame version
does your execution succeeded or failed
l
Len Strnad
07/11/2023, 8:45 PMSucceeds.
There are some subworkflows, which might be causing the task_mode_metadata to be None?
j
Jay Ganbat
07/11/2023, 8:46 PMoh its not task but subworkflow?
l
Len Strnad
07/11/2023, 8:47 PMexecution = remote.fetch_execution(name="ffc0e12bb3e314fae9aa")The nodes in
Copy code
{'end-node': 0.0,
'n0': 2.698262133333333,
'n1': 2.202614066666667,
'n2': 1.59931375,
'n3': 1.8271749,
'n4': 30.83562241666667,
'n5': 0.2830597833333333,
'n6': 0.3337019666666666,
'start-node': 0.0}But that is the depth I would like to achieve.
j
Jay Ganbat
07/11/2023, 8:49 PMso in JH, i just print the entire execution and in there all not None attributes are displayed
jupyterhub
l
Len Strnad
07/11/2023, 8:53 PMInteresting. My case is a bit weird, but my workflow is a single subworkflow containing subworkflows and tasks lol. These subworkflows and tasks are the level at which I would like the duration captured.
When I print the execution, I see the name of the subworkflow only.
j
Jay Ganbat
07/11/2023, 8:58 PMyeah without seeing some example or code im all out of ideas haha
l
Len Strnad
07/11/2023, 9:07 PMNo worries! Thanks for jumping in and helping!
I think I might be on to something. I will report back
This gets me one step closer, but only includes tasks and not subworkflows:
Copy code
from flytekit.remote import FlyteRemote
from flytekit.models.core.identifier import NodeExecutionIdentifier
from flytekit.clients.helpers import iterate_task_executions
from flytekit.configuration import Config
remote = FlyteRemote(
config=Config.for_endpoint(endpoint=endpoint),
default_project=project,
default_domain=domain,
)
execution = remote.fetch_execution(name="ffc0e12bb3e314fae9aa")
execution = remote.sync_execution(execution=execution, sync_nodes=True)
profiles = {}
for node_execution in execution.node_executions['n0'].executions:
nid = NodeExecutionIdentifier(node_id=node_execution.id.node_id, execution_id=execution.id)
for t in iterate_task_executions(client=remote.client, node_execution_identifier=nid):
profiles[t.id.task_id.name] = t.closure.durationj
Jay Ganbat
07/11/2023, 9:20 PMwere you able to get the execution id of the subworkflow? last time i tried it was not possible
l
Len Strnad
07/11/2023, 11:10 PMI think I am managing to do this is this snipped below, granted it is not very elegant or tested:
Copy code
from flytekit.remote import FlyteRemote
from flytekit.models.core.identifier import NodeExecutionIdentifier
from flytekit.clients.helpers import iterate_task_executions
from flytekit.configuration import Config
remote = FlyteRemote(
config=Config.for_endpoint(endpoint=endpoint),
default_project=project,
default_domain=domain,
)
execution = remote.fetch_execution(name="ffc0e12bb3e314fae9aa")
execution = remote.sync(execution=execution, sync_nodes=True)
profiles = {}
def nested_task_info(node_executions):
for ne in node_executions.values():
if ne.id.node_id == "start-node" or ne.id.node_id == "end-node":
continue
if ne.metadata.is_parent_node:
nested_task_info(ne.subworkflow_node_executions)
else:
nid = NodeExecutionIdentifier(node_id=ne.id.node_id, execution_id=execution.id)
for t in iterate_task_executions(client=remote.client, node_execution_identifier=nid):
profiles[t.id.task_id.name] = t.closure.duration
nested_task_info(execution.node_executions)j
Jay Ganbat
07/11/2023, 11:54 PMawesome, i will give it go on my workflows
k
Ketan (kumare3)
07/12/2023, 6:56 AM
Cc @Dan Rammer (hamersaw) - there is new api that returns the timeline -
l
Len Strnad
07/12/2023, 1:16 PM@Ketan (kumare3) does this exist today or is this in the works?
k
Ketan (kumare3)
07/12/2023, 9:55 PM
Yes
l
Len Strnad
07/12/2023, 9:56 PMI couldn’t find anything. If you could share more details that would be appreciated.
d
Dan Rammer (hamersaw)
07/13/2023, 1:30 AM@Len Strnad I believe we're referring to the runtime metrics work I recently did. It's used to fuel the timeline breakdown in the UI (attached image). A good example of this API is the
pyflyte metrics explainl
Len Strnad
07/13/2023, 1:11 PMAwesome!
2 Views