Hi, is it possible to get at some underlying task execution data from the flytekit python sdk? Specifically, say I create a

SagemakerBuiltinAlgorithmsTask

and I would like to retrieve the

TrainingJobName

, which the backend plugin has access to.

Hi, @thankful-dress-89577! You can try calling

flytekit.current_context()

within a task to get some of the metadata. You can find all the acceptable parameters here. I believe the

TrainingJobName

you’re referring to is the pod name pertaining to a task. @glamorous-carpet-83516, can we retrieve this at the Python end?

Could you try flytekit remote to fetch execution? Metadata (like job name) should be included in the task execution. https://github.com/flyteorg/flytekit/blob/571fc89013ad9506d283a9ed3d0a88de6004a064/flytekit/remote/remote.py#L284-L305

We couldn’t get this working from the python sdk for some reason, but via flytectl we are able to do:

flytectl get execution --project myproject -d development abcdefhv5l8d8bgdpxcb -o yaml

but that requires already knowing most of the job name, the actual job name shows up like

abcdefhv5l8d8bgdpxcb-n1-0

Interestingly, flyte console seems to know it and provides us a nice link right to sagemaker training job console.

We don’t yet provide a way to identify a task execution uniquely; here’s the issue: https://github.com/flyteorg/flyte/issues/2558. You could, however, retrieve

current_context().execution_id

and

current_context().task_id

, but this doesn’t uniquely identify two task executions within the same workflow.

OK, I see. I think we have found a workaround for custom sagemaker training tasks -> we can retrieve the training job name from the environment variable set by sagemaker and pass it as a task output. For builtin-algorithms, we don’t get to customize the code or task outputs so we cannot yet retrieve that job name.