Hello flyte community I m facing the following bug I ve defi

Question

Hello flyte community I m facing the following bug I ve defined a task function ``` task def to training config cfg DictConfig gt TrainingConfig return TrainingConfig cfg ``` `TrainingConfig` is simply a custom `dataclass json` object This function simply aims to convert a `DictConfig` object to the dataclass so I can reuse it later within my workflow I am getting the following error ``` AttributeError Traceback most recent call last opt venv lib python3 7 site packages flytekit core base task py in dispatch execute self ctx input literal map 521 try gt 522 literals k = TypeEngine to literal exec ctx v py type literal type 523 except Exception as e opt venv lib python3 7 site packages flytekit core type engine py in to literal cls ctx python val python type expected 751 gt 752 lv = transformer to literal ctx python val python type expected 753 opt venv lib python3 7 site packages flytekit core type engine py in to literal self ctx python val python type expected 349 gt 350 self serialize flyte type python val python type 351 return Literal opt venv lib python3 7 site packages flytekit core type engine py in serialize flyte type self python val python type 397 field type = v type gt 398 python val setattr v name self serialize flyte type val field type 399 return python val opt venv lib python3 7 site packages flytekit core type engine py in serialize flyte type self python val python type 395 for v in dataclasses fields python type gt 396 val = python val getattribute v name 397 field type = v type AttributeError DictConfig object has no attribute name ``` `Note` this works perfectly fine when I remove the ` task` decorator Any idea what could be the cause

Ketan (kumare3) · Answer

Interesting

Ketan (kumare3) · Answer

What s a dictconfig

Kilian Tep · Answer

It s from the omegaconf library It s basically a more rigid dictionary which is quite useful for configurations <https omegaconf readthedocs io en 2 0 branch usage html|https omegaconf readthedocs io en 2 0 branch usage html>

Ketan (kumare3) · Answer

Interesting we should add a type transformer for it

Ketan (kumare3) · Answer

Then we won t need to change it at all It seems It is being pickled

Kilian Tep · Answer

so the pickling is what s causing the error right

Ketan (kumare3) · Answer

Not sure but a guess

Ketan (kumare3) · Answer

Could also be the dataclass json decorator screwing things uo

Kilian Tep · Answer

so fun fact i added the `dataclass json` decorator because it raised the following error before I did ``` AssertionError Traceback most recent call last tmp ipykernel 31113 2006354963 py in lt module gt gt 1 from global search training tasks config import get generic config print training config 2 from global search training schema super schema import TrainingConfig to training config 3 from omegaconf import DictConfig OmegaConf workspaces global search training global search training tasks config py in lt module gt 4 from flytekit import task 5 from hydra core global hydra import GlobalHydra gt 6 from global search training schema super schema import TrainingConfig 7 from omegaconf import DictConfig OmegaConf 8 from hydra core config store import ConfigStore workspaces global search training global search training schema super schema py in lt module gt 18 19 task gt 20 def to training config cfg DictConfig gt TrainingConfig 21 return TrainingConfig cfg 22 opt venv lib python3 7 site packages flytekit core task py in task task function task config cache cache serialize cache version retries interruptible deprecated timeout container image environment requests limits secret requests execution mode task resolver disable deck 210 211 if task function gt 212 return wrapper task function 213 else 214 return wrapper opt venv lib python3 7 site packages flytekit core task py in wrapper fn 204 execution mode=execution mode 205 task resolver=task resolver gt 206 disable deck=disable deck 207 208 update wrapper task instance fn opt venv lib python3 7 site packages flytekit core tracker py in call cls args kwargs 33 34 def call cls args kwargs gt 35 o = super InstanceTrackingMeta cls call args kwargs 36 o instantiated in = InstanceTrackingMeta find instance module 37 return o opt venv lib python3 7 site packages flytekit core python function task py in init self task config task function task type ignore input vars execution mode task resolver kwargs 125 task config=task config 126 task resolver=task resolver gt 127 kwargs 128 129 opt venv lib python3 7 site packages flytekit core python auto container py in init self name task config task type container image requests limits environment task resolver secret requests kwargs 76 task config=task config 77 security ctx=sec ctx gt 78 kwargs 79 80 self container image = container image opt venv lib python3 7 site packages flytekit core base task py in init self task type name task config interface environment disable deck kwargs 384 task type=task type 385 name=name gt 386 interface=transform interface to typed interface interface 387 kwargs 388 opt venv lib python3 7 site packages flytekit core interface py in transform interface to typed interface interface 218 219 inputs map = transform variable map interface inputs input descriptions gt 220 outputs map = transform variable map interface outputs output descriptions 221 return interface models TypedInterface inputs map outputs map 222 opt venv lib python3 7 site packages flytekit core interface py in transform variable map variable map descriptions 326 if variable map 327 for k v in variable map items gt 328 res k = transform type v descriptions get k k 329 sub type Type T = v 330 if hasattr v origin and hasattr v args opt venv lib python3 7 site packages flytekit core interface py in transform type x description 344 345 def transform type x type description str = None gt interface models Variable gt 346 return interface models Variable type=TypeEngine to literal type x description=description 347 348 opt venv lib python3 7 site packages flytekit core type engine py in to literal type cls python type 708 709 transformer = cls get transformer python type gt 710 res = transformer get literal type python type 711 data = None 712 if get origin python type is Annotated opt venv lib python3 7 site packages flytekit core type engine py in get literal type self t 317 if not issubclass t DataClassJsonMixin 318 raise AssertionError gt 319 f Dataclass t should be decorated with dataclass json to be f serialized correctly 320 321 schema = None AssertionError Dataclass lt class global search training schema super schema TrainingConfig gt should be decorated with dataclass json to be serialized correctly```

Kilian Tep · Answer

interesting bugs sweat smile

Samhita Alla · Answer

Not sure why the field is of type `DictConfig` What s the output of `TrainingConfig cfg ` generally

Kilian Tep · Answer

< Samhita Alla> I tried to change the field to `dict` but the error became ```AttributeError dict object has no attribute name ```

Kilian Tep · Answer

Let s say I define a class like this ``` dataclass json dataclass class TrainingConfig environment str``` Then I would declare the object by doing ```cfg = enviromnent staging training config = TrainingConfig cfg ``` output should be like ```TrainingConfig enviroment= staging ```

Samhita Alla · Answer

Yeah Let me repro it

Samhita Alla · Answer

```from dataclasses import dataclass from dataclasses json import dataclass json from flytekit import task workflow from omegaconf import DictConfig OmegaConf dataclass json dataclass class TrainingConfig environment str task def t1 cfg DictConfig gt TrainingConfig return TrainingConfig cfg workflow def wf cfg DictConfig = OmegaConf create environment staging t1 cfg=cfg wf ``` This works for me Isn t this what you re trying to do Is it working locally for you

Kilian Tep · Answer

ok sorry < Samhita Alla> my bad for not framing the problem exactly the same I think the problem comes from the fact that I have nested classes Here s something more similar to my case where I can reproduce the bug ```from dataclasses import dataclass from dataclasses json import dataclass json from flytekit import task workflow from omegaconf import DictConfig OmegaConf from global search training schema primary schema import GCPConfig dataclass json dataclass class TrainingConfig environment str gcp str dataclass json dataclass class NestedClass name str training TrainingConfig task def t1 cfg DictConfig gt NestedClass return NestedClass cfg workflow def wf cfg DictConfig = OmegaConf create name staging training environment staging gcp test t1 cfg=cfg wf ```

Kilian Tep · Answer

```AttributeError Traceback most recent call last opt venv lib python3 7 site packages flytekit core base task py in dispatch execute self ctx input literal map 521 try gt 522 literals k = TypeEngine to literal exec ctx v py type literal type 523 except Exception as e opt venv lib python3 7 site packages flytekit core type engine py in to literal cls ctx python val python type expected 751 gt 752 lv = transformer to literal ctx python val python type expected 753 opt venv lib python3 7 site packages flytekit core type engine py in to literal self ctx python val python type expected 349 gt 350 self serialize flyte type python val python type 351 return Literal opt venv lib python3 7 site packages flytekit core type engine py in serialize flyte type self python val python type 397 field type = v type gt 398 python val setattr v name self serialize flyte type val field type 399 return python val opt venv lib python3 7 site packages flytekit core type engine py in serialize flyte type self python val python type 395 for v in dataclasses fields python type gt 396 val = python val getattribute v name 397 field type = v type AttributeError DictConfig object has no attribute environment ```

Kilian Tep · Answer

so the bug is slightly different so this seems to indicate that it s having issues fetching the key `environment` Above it was ```AttributeError DictConfig object has no attribute name ``` because `TrainingConfig` is a nested config and one of the dataclasses inside had an attribute called `name`

Samhita Alla · Answer

```from dataclasses import dataclass from dataclasses json import dataclass json from flytekit import task workflow from omegaconf import DictConfig OmegaConf dataclass json dataclass class TrainingConfig environment str gcp str dataclass json dataclass class NestedClass name str training TrainingConfig task def t1 cfg DictConfig gt NestedClass return NestedClass cfg name TrainingConfig cfg training workflow def wf cfg DictConfig = OmegaConf create name staging training environment staging gcp test t1 cfg=cfg wf ```

Samhita Alla · Answer

This works I think the error s cropping up because a dataclass needs to be initialized before sending the arguments

Kilian Tep · Answer

I see Is it because flyte works with promise objects Is it comparable to lazy evaluation in a way

Kilian Tep · Answer

Ok thanks so much < Samhita Alla> literally been stuck on this for almost 2 days

Samhita Alla · Answer

gt I see Is it because flyte works with promise objects Is it comparable to lazy evaluation in a way It isn t because of Promises If you see the following code ```from dataclasses import dataclass from dataclasses json import dataclass json from flytekit import task workflow from omegaconf import DictConfig OmegaConf dataclass json dataclass class TrainingConfig environment str gcp str dataclass json dataclass class NestedClass name str training int cfg DictConfig = OmegaConf create name staging training environment staging gcp test print NestedClass cfg ``` This works despite `training` type set to `int` cause type validation isn t done in this case In the Flyte case the types are validated at every step and hence you were seeing the error