Does anyone know how to search through runs and outputs in python? I for some reason cannot figure out how to do that. I am trying to see how much metadata I can parse through in python. Thanks for your help!

hey @Evan Sadler are you using FlyteRemote? https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#design-control-plane

@Niels Bantilan that looks exactly like what I am looking for! Thank you

yep, the cluster you get access to in

FlyteRemote

depends on the

config

argument.

Config.auto()

will do this to try and automatically find the config file, but you can use

Config.for_sandbox()

for a

flytectl sandbox

cluster and

Config.for_endpoint()

for a specific endpoint identified via dns

Yes, it should work.

Awesome and thank you all for the information! I will give it a shot. For context some people are talking about ML Metadata (MLMD) and like tracking metadata. Flyte inherently does this, the question is what surfacing the information looks like and will be pretty straight forward to investigate.

Flyte inherently does this, the question is what surfacing the information looks like and will be pretty straight forward to investigate

yes! we’ve been trying to figure out what kinds of utility functions to provide… for example, if we had:

remote = FlyteRemote(...)
remote.fetch_executions_with_output_type(sklearn.base.BaseEstimator)

would get all executions with sklearn models as outputs. If you could articulate your different use cases, we’d love to figure out what those utility functions would be!

That blew my mind. I will talk to @Kenneth Sylvain @Cyrus and see if there are requirements anywhere.

So assuming the execution is a workflow:

execution = remote.fetch_execution(...)

# make sure all the nodes are synced
synced_execution = remote.sync(execution, sync_nodes=True)

# get the node execution inputs/outputs
synced_execution.node_executions["n1"].inputs
synced_execution.node_executions["n1"].outputs["model_file"]

https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#retrieving-inspecting-executions The you can get all the node keys with

synced_execution.node_executions.keys()

I forget now how the node execution keys are named… but that should allow you to get input/output data from tasks / subworkflow executions

Exactly what I needed 🙂