https://flyte.org
Join Slack
Hi guys, can I combine flyte workflow with hydra?
# flyte-support
n
Hi guys, can I combine flyte workflow with hydra?
a
Hi @nutritious-addition-55433 You can combine it yes, @cool-lifeguard-49380 previously shared their experience using both systems:

https://www.youtube.com/watch?v=tghvVvHJi7s▾

n
Nice! thank you fir the information
f
is there a reference implementation of that integration? Is typing integration code from youtube video frame the only option?
c
I’m not working at this company anymore but I think unfortunately it’s not yet open sourced 😕 I’ll ping former colleagues to see if they can
f
thanks @cool-lifeguard-49380
I was able to make hydra config adapter work
Copy code
from typing import Type, Optional

from flyteidl.core.types_pb2 import SimpleType
from flytekit import LiteralType, FlyteContext, Literal, Scalar
from flytekit.extend import TypeTransformer, T
from google.protobuf.json_format import MessageToDict
from omegaconf import DictConfig, OmegaConf
from google.protobuf.struct_pb2 import Struct

class DictConfigTransformer(TypeTransformer[DictConfig]):
    _TYPE_INFO = LiteralType(simple=SimpleType.STRUCT)

    def to_literal(self, ctx: FlyteContext, python_val: DictConfig, python_type: Type[DictConfig],
                   expected: LiteralType) -> Literal:
        s = Struct()
        conf = OmegaConf.to_container(python_val)
        s.update(conf)
        return Literal(scalar=Scalar(generic=s))

    def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[DictConfig]) -> Optional[T]:
        message = lv.scalar.generic
        return OmegaConf.create(MessageToDict(message))

    def get_literal_type(self, t: Type[T]) -> LiteralType:
        return Type[DictConfig]
but I am struggling with the launcher
core functions I see in the video are missing from flytekit:
Copy code
tasks, workflow, launch_plan = flyte_core.extract_flyte_entities_from_module(self.config.module_name)
c
@delightful-greece-6207 @thankful-flag-82043 fyi, would you be willing to share this?
t
we plan to make it open source @delightful-greece-6207 can provide more insights on the timeline
f
thanks @thankful-flag-82043, that would be super helpful. I am currently trying to extract minimal set of functions from flyte-zen to make it work.
@delightful-greece-6207, please share any information on the timeline
d
@flaky-beach-18445 Our setup consists of two parts: a TypeTransformer for DictConfigs and a HydraLauncher. Capacity allowing, I will try and add the TypeTransformer to the flytekit next week. The HydraLauncher will require a little polish and I will try and add that early next year.
gratitude thank you 2
🙌 1
🙌🏽 1
🙇🏽 1
f
meanwhile, I am exploring an option to pass around dataclasses (instead of 
DictConf
as proposed in the video) • in the main parse DictConf into a dataclass object, which is the same config represented as a dataclass • pass dataclass object as an argument to the workflow That elimitanes the need to deal with 
DictConf
serialization.
Copy code
@dataclass_json
@dataclass
class Configuration:
    trainer: TrainerConfig
    pruner: PrunerConfig
    flyte_environment: FlyteEnvironment

@workflow
def finetune_workflow(config: Configuration) -> None:
    train(config=config)

@hydra.main(version_base=None, config_path='conf', config_name="default")
def main(cfg) -> None:
    config = instantiate(cfg, _convert_="none")
    finetune_workflow(config=config)
26 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by