https://flyte.org logo
Join Slack
Powered by
Hi guys, can I combine flyte workflow with hydra?
# ask-the-community
m
Hi guys, can I combine flyte workflow with hydra?
d
Hi @Muhammad Haritsah Mukhlis You can combine it yes, @Fabio Grätz previously shared their experience using both systems:

https://www.youtube.com/watch?v=tghvVvHJi7s▾

m
Nice! thank you fir the information
m
is there a reference implementation of that integration? Is typing integration code from youtube video frame the only option?
f
I’m not working at this company anymore but I think unfortunately it’s not yet open sourced 😕 I’ll ping former colleagues to see if they can
m
thanks @Fabio Grätz
I was able to make hydra config adapter work
Copy code
from typing import Type, Optional

from flyteidl.core.types_pb2 import SimpleType
from flytekit import LiteralType, FlyteContext, Literal, Scalar
from flytekit.extend import TypeTransformer, T
from google.protobuf.json_format import MessageToDict
from omegaconf import DictConfig, OmegaConf
from google.protobuf.struct_pb2 import Struct

class DictConfigTransformer(TypeTransformer[DictConfig]):
    _TYPE_INFO = LiteralType(simple=SimpleType.STRUCT)

    def to_literal(self, ctx: FlyteContext, python_val: DictConfig, python_type: Type[DictConfig],
                   expected: LiteralType) -> Literal:
        s = Struct()
        conf = OmegaConf.to_container(python_val)
        s.update(conf)
        return Literal(scalar=Scalar(generic=s))

    def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[DictConfig]) -> Optional[T]:
        message = lv.scalar.generic
        return OmegaConf.create(MessageToDict(message))

    def get_literal_type(self, t: Type[T]) -> LiteralType:
        return Type[DictConfig]
but I am struggling with the launcher
core functions I see in the video are missing from flytekit:
Copy code
tasks, workflow, launch_plan = flyte_core.extract_flyte_entities_from_module(self.config.module_name)
f
@Sebastian Schulze @Thomas Wollmann fyi, would you be willing to share this?
t
we plan to make it open source @Sebastian Schulze can provide more insights on the timeline
m
thanks @Thomas Wollmann, that would be super helpful. I am currently trying to extract minimal set of functions from flyte-zen to make it work.
@Sebastian Schulze, please share any information on the timeline
s
@Maksim Khadkevich Our setup consists of two parts: a TypeTransformer for DictConfigs and a HydraLauncher. Capacity allowing, I will try and add the TypeTransformer to the flytekit next week. The HydraLauncher will require a little polish and I will try and add that early next year.
m
meanwhile, I am exploring an option to pass around dataclasses (instead of 
DictConf
as proposed in the video) • in the main parse DictConf into a dataclass object, which is the same config represented as a dataclass • pass dataclass object as an argument to the workflow That elimitanes the need to deal with 
DictConf
serialization.
Copy code
@dataclass_json
@dataclass
class Configuration:
    trainer: TrainerConfig
    pruner: PrunerConfig
    flyte_environment: FlyteEnvironment

@workflow
def finetune_workflow(config: Configuration) -> None:
    train(config=config)

@hydra.main(version_base=None, config_path='conf', config_name="default")
def main(cfg) -> None:
    config = instantiate(cfg, _convert_="none")
    finetune_workflow(config=config)
2 Views
Open in Slack
PreviousNext
Powered by