Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello, I’m wondering how involved it would be to extend the sagemaker plugin to support additional task types like batch transforms and creating/updating endpoint configurations. Is this something that can be done purely in python (flytekit)?

it can be done in flytekit (well everything can be), but the backend plugins are very efficient

but, thats a trade-off. I think you should always start with a flytekit only plugin and once happy we can help you make a backend plugin

Also if you like the interface lets share it in the community sync and there are lot of folks who can help with the backend :slightly_smiling_face:

So with the python only approach would you suggest to use boto3 or sagemaker sdk directly?

Going to piggyback off this q... if we wanted to extend the sagemaker training job plugin, would PRs only be needed for these files?
• <https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/plugins/sagemaker/training_job.proto>
• <https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker/models/training_job.py>

cc <@U03K6Q3CLUE> / <@U03LB7964RK> we at Union would love to help communicate. But most of us are our for the next few days. How about 7th / 8th?

<@U03K6Q3CLUE> you are absolutely right. But, the problem with sagemaker plugin today is the dependence on the sagemaker operator

We wanted to eventually just migrate away from the Sagemaker operator to use - Flyte’s WEB Api interface, makes it much easier to manage connections without the dependence on the operator, which is very stale

last update 2 years ago - <https://github.com/aws/amazon-sagemaker-operator-for-k8s>

<@UNZB4NW3S> Returning to this thread, I think I stumbled into an issue with using Flyte’s Sagemaker training integration that I’m curious if you’ve resolved. As it stands, it doesn’t seem possible to associate a training job with a sagemaker experiment, so it wouldn’t be possible to use SM Studio UI to compare trials or otherwise view training metrics. What kind of pattern makes the most sense to you in terms of having experiment tracking and comparisons on top of training orchestrated by Flyte?

Hey Andrew would you be open for a quick chat

<@UNZB4NW3S> yes I am available today to chat