Hello What s the best way to define a task config that launc Flyte #announcements

Hello! What’s the best way to define a task config...

Nada Saiyed

08/05/2022, 12:41 AM

Hello! What’s the best way to define a task config that launches a custom SageMaker hyperparameter tuning job. I am trying to create a task similar to this example. i see i can provide the

task_config

as a

HPOJob

but how do i specify which

training_task

it’s supposed to execute? and also where should i specify the parameters like

HyperparameterTuningJobConfig

, and

ParameterRanges

Nada Saiyed

08/05/2022, 12:46 AM

Also, how will different values of hyperparameters be passed into the custom train script? will i need to read it from SageMaker’s

hyperparameter.json

Samhita Alla

08/05/2022, 6:35 AM

Hi, @Nada Saiyed! If you already have a training task defined, be it a custom or a builtin training task, you can send that task as the

training_task

SagemakerHPOTask

. You can send

HyperparameterTuningJobConfig

and

ParameterRanges

as inputs to a HPO task when executing it, as can be seen here. Here’s how the task inputs would render on the UI:

Samhita Alla

08/05/2022, 6:36 AM

Also, how will different values of hyperparameters be passed into the custom train script?

I guess you can send them as inputs to it directly? Like how you send to a Flyte task? Lemme know if I misunderstood your question.

Nada Saiyed

08/05/2022, 2:20 PM

can i define the task as a python function? E.g.

Copy code

@task(
    task_config=HPOJob(
        max_number_of_training_jobs=3,
        max_parallel_training_jobs=2,
        tunable_params=["num_round", "max_depth", "gamma"],
    ),

)
def my_hpo_task(x:int):
    print(x)

in this case where do i define

HyperparameterTuningJobConfig

and

ParameterRanges

Nada Saiyed

08/05/2022, 2:22 PM

and i have my training task defined like this:

Copy code

@task(
    task_config=SagemakerTrainingJobConfig(
        algorithm_specification=AlgorithmSpecification(
            input_mode=InputMode.FILE,
            algorithm_name=AlgorithmName.CUSTOM,
            algorithm_version="",
            metric_definitions = [MetricDefinition(name="score", regex="score: ([0-9\\.]+)")],
            input_content_type=InputContentType.TEXT_CSV,
        ),
        training_job_resource_config=TrainingJobResourceConfig(
            instance_type="ml.m4.4xlarge",
            instance_count=1,
            volume_size_in_gb=25,
        ),
    ),
    interruptible=True
)
def custom_training_task(x:int):
    print(x)

so in this case will the

training_task

custom_training_task

Nada Saiyed

08/05/2022, 3:26 PM

also how can i use this task as part of a workflow?

Nada Saiyed

08/05/2022, 3:37 PM

@katrina any thoughts?

Nada Saiyed

08/05/2022, 3:59 PM

Also what inputs does this task expect? in the last line on the page you linked it says

inputs=hpo_inputs

what are

hpo_inputs

Nada Saiyed

08/05/2022, 5:27 PM

i was able to create a HPO task with hyperparameters as inputs, but the task fails to launch a SageMaker HPO Job with this error msg:

Copy code

[BadTaskSpecification] Error occurred when checking if all the required inputs exist, caused by: [SAGEMAKER_ERROR] Required input not specified: [train]

This is one of the parameters that

SagemakerBuiltinAlgorithmsTask

expects, but i am working with a

SagemakerCustomTrainingTask

, not sure why its expecting these builtIn parameters.

Nada Saiyed

08/05/2022, 5:28 PM

i also tried giving dummy values to these “expected” parameters i.e.

static_hyperparameters

train

validation

but ofcourse it failed with incorrect type

katrina

08/05/2022, 5:30 PM

hey sorry can you summarize what you're running into here?

Andrew Achkar

08/05/2022, 5:41 PM

I think we’re running into basically HPO + Custom Sagemaker Training is not implemented. The error above is coming from here: https://github.com/flyteorg/flyteplugins/blob/v1.0.8/go/tasks/plugins/k8s/sagemaker/hyperparameter_tuning.go#L68-L73 It is checking for are only the inputs defined by a builtin sagemaker algorithm. That’s fairly confusing because the flytekit code suggests both are supported but the backend seems to not have this yet. Here’s an old PR that aims to add that support: https://github.com/flyteorg/flyteplugins/pull/137

Andrew Achkar

08/05/2022, 5:55 PM

This init signature and validation logic made me think both types should be supported: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker/hpo.py#L48-L58

Samhita Alla

08/06/2022, 1:01 PM

OH NO! @Ketan (kumare3), any idea why we haven’t merged https://github.com/flyteorg/flyteplugins/pull/137?

Ketan (kumare3)

08/06/2022, 2:30 PM

This is because at lyft we decided to not use this as it was not a great api, but that being said we can help here. So IIRC the way it works - you need to send metrics from custom training task which are sent to cloudwatch and then it should work.

Ketan (kumare3)

08/06/2022, 2:31 PM

@Andrew Achkar / @Nada Saiyed this is terrible experience and we are extremely sorry about that

Ketan (kumare3)

08/06/2022, 2:32 PM

We also don't implement things that people don't use, they easily go stale

Ketan (kumare3)

08/06/2022, 2:32 PM

If you folks will use it we can implement this

Ketan (kumare3)

08/06/2022, 3:05 PM

@Samhita Alla let's merge this one - https://github.com/flyteorg/flytekit/pull/1112. Cc @Yee

✅ 1

merged 1

160 Views

Open in Slack

Previous Next