Thread
#ask-the-community
    a

    Andrew Achkar

    2 months ago
    Hi, is it possible to get at some underlying task execution data from the flytekit python sdk? Specifically, say I create a
    SagemakerBuiltinAlgorithmsTask
    and I would like to retrieve the
    TrainingJobName
    , which the backend plugin has access to.
    Samhita Alla

    Samhita Alla

    1 month ago
    Hi, @Andrew Achkar! You can try calling
    flytekit.current_context()
    within a task to get some of the metadata. You can find all the acceptable parameters here. I believe the
    TrainingJobName
    you’re referring to is the pod name pertaining to a task. @Kevin Su, can we retrieve this at the Python end?
    Kevin Su

    Kevin Su

    1 month ago
    Could you try flytekit remote to fetch execution? Metadata (like job name) should be included in the task execution. https://github.com/flyteorg/flytekit/blob/571fc89013ad9506d283a9ed3d0a88de6004a064/flytekit/remote/remote.py#L284-L305
    a

    Andrew Achkar

    1 month ago
    We couldn’t get this working from the python sdk for some reason, but via flytectl we are able to do:
    flytectl get execution --project myproject -d development abcdefhv5l8d8bgdpxcb -o yaml
    but that requires already knowing most of the job name, the actual job name shows up like
    abcdefhv5l8d8bgdpxcb-n1-0
    Interestingly, flyte console seems to know it and provides us a nice link right to sagemaker training job console.
    Oh and
    flytectl get execution
    doesn’t include the actual
    TrainingJobName
    . I don’t see the custom_info of the TaskExecutionClosure. In the python representation too I do not see a custom_info attribute: https://github.com/flyteorg/flytekit/blob/master/flytekit/models/admin/task_execution.py#L8
    I think the above is essentially just the TaskExecutionIdentifier. Asked another way, is there a way to know the current TaskExecutionIdentifier within a task?
    Samhita Alla

    Samhita Alla

    1 month ago
    We don’t yet provide a way to identify a task execution uniquely; here’s the issue: https://github.com/flyteorg/flyte/issues/2558. You could, however, retrieve
    current_context().execution_id
    and
    current_context().task_id
    , but this doesn’t uniquely identify two task executions within the same workflow.
    a

    Andrew Achkar

    1 month ago
    OK, I see. I think we have found a workaround for custom sagemaker training tasks -> we can retrieve the training job name from the environment variable set by sagemaker and pass it as a task output. For builtin-algorithms, we don’t get to customize the code or task outputs so we cannot yet retrieve that job name.