https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Andrew Achkar

07/28/2022, 5:54 PM
Hi, is it possible to get at some underlying task execution data from the flytekit python sdk? Specifically, say I create a
SagemakerBuiltinAlgorithmsTask
and I would like to retrieve the
TrainingJobName
, which the backend plugin has access to.
s

Samhita Alla

07/29/2022, 5:23 AM
Hi, @Andrew Achkar! You can try calling
flytekit.current_context()
within a task to get some of the metadata. You can find all the acceptable parameters here. I believe the
TrainingJobName
you’re referring to is the pod name pertaining to a task. @Kevin Su, can we retrieve this at the Python end?
k

Kevin Su

07/29/2022, 6:16 AM
Could you try flytekit remote to fetch execution? Metadata (like job name) should be included in the task execution. https://github.com/flyteorg/flytekit/blob/571fc89013ad9506d283a9ed3d0a88de6004a064/flytekit/remote/remote.py#L284-L305
a

Andrew Achkar

07/29/2022, 9:21 PM
We couldn’t get this working from the python sdk for some reason, but via flytectl we are able to do:
Copy code
flytectl get execution --project myproject -d development abcdefhv5l8d8bgdpxcb -o yaml
but that requires already knowing most of the job name, the actual job name shows up like
abcdefhv5l8d8bgdpxcb-n1-0
Interestingly, flyte console seems to know it and provides us a nice link right to sagemaker training job console.
Oh and
flytectl get execution
doesn’t include the actual
TrainingJobName
. I don’t see the custom_info of the TaskExecutionClosure. In the python representation too I do not see a custom_info attribute: https://github.com/flyteorg/flytekit/blob/master/flytekit/models/admin/task_execution.py#L8
I think the above is essentially just the TaskExecutionIdentifier. Asked another way, is there a way to know the current TaskExecutionIdentifier within a task?
s

Samhita Alla

08/03/2022, 4:59 AM
We don’t yet provide a way to identify a task execution uniquely; here’s the issue: https://github.com/flyteorg/flyte/issues/2558. You could, however, retrieve
current_context().execution_id
and
current_context().task_id
, but this doesn’t uniquely identify two task executions within the same workflow.
a

Andrew Achkar

08/03/2022, 2:00 PM
OK, I see. I think we have found a workaround for custom sagemaker training tasks -> we can retrieve the training job name from the environment variable set by sagemaker and pass it as a task output. For builtin-algorithms, we don’t get to customize the code or task outputs so we cannot yet retrieve that job name.
28 Views