https://flyte.org logo
Join Slack
Powered by
New RunLLM response generated: >>>*# Question* he...
# runllm-flyte-trial
a
New RunLLM response generated: >>>*# Question* hello, I am trying to work through an issue with using 
dataclass
when I'm running a workflow, and I need some help. I get this error when the tasks/workflow are being compiled: 
TypeError: Invalid base type typing.Union in call to isinstance
. This workflow ran fine, until I started using a 
dataclass
that I defined. To figure out the problem, I decided to use the 
dataclass.py
example from here. I'm running everything locally (ideally without the Flyte demo cluster) to keep things as simple as possible. I noticed that if I run the example with 
pyflyte run dataclass.py dataclass_wf --x 10 --y 20
, the workflow runs fine and outputs the result. OTOH, if I run it with 
python dataclass.py
(which I assume should work because of the 
if ___name___ == "main"
directive at the bottom of 
dataclass.py
, I get a large number of errors that seem to either be 
TypeError: Invalid base type typing.Union in call to isinstance
or 
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union
. For context: the reason I'm trying to run my Flyte workflow via the main method is because I'm using hydra and I could not figure out another way to make sure that the Hydra configs are loaded and made available to the program. I want able to write my workflow so it worked with Omegaconf, but ran into this issue when I introduced dataclasses. so I am wondering: 1. Is it really only possible to run Flyte workflows via 
pyflyte
? 2. Should I not expect things to work if I run a Flyte workflow through a file's main function? 3. How can I write my workflow so that I can use dataclasses and I can populate Hydra configs (or somewhat equivalently, run the workflow via the file's main function)? # Answer 1. Running Flyte Workflows: Flyte workflows can be run both via 
pyflyte run
and directly through a file's main function. However, there are some nuances to be aware of, especially when dealing with dataclasses and type annotations. 2. Running via Main Function: Running a Flyte workflow through a file's main function should work, but issues can arise due to type handling, especially with 
typing.Union
and dataclasses. The error you're encountering (
TypeError: Invalid base type typing.Union in call to isinstance
) suggests that Flyte's type system is having trouble with the 
Union
type in your dataclass. 3. Using Dataclasses and Hydra: To use dataclasses and populate Hydra configs, you need to ensure that the dataclass types are correctly handled by Flyte's type system. Flyte uses the Mashumaro library for serializing and deserializing dataclasses, and there are known issues with handling 
Union
types in dataclasses. Example Workflow with Dataclass and Hydra
>>Here's an example of how you can structure your workflow to use dataclasses and Hydra:
>>```
import os import tempfile from dataclasses import dataclass from flytekit import task, workflow from flytekit.types.directory import FlyteDirectory from flytekit.types.file import FlyteFile from flytekit.types.structured import StructuredDataset from mashumaro.mixins.json import DataClassJSONMixin from hydra import initialize, compose @dataclass class Datum(DataClassJSONMixin): x: int y: str z: dict[int, str] @task def stringify(s: int) -> Datum: return Datum(x=s, y=str(s), z={s: str(s)}) @task def add(x: Datum, y: Datum) -> Datum: x.z.update(y.z) return Datum(x=x.x + y.x, y=x.y + y.y, z=x.z) @workflow def dataclass_wf(x: int, y: int) -> Datum: o1 = add(x=stringify(s=x), y=stringify(s=y)) return o1 if name == "__main__": # Initialize Hydra with initialize(config_path="path/to/configs"): cfg = compose(config_name="config") x = cfg.x y = cfg.y result = dataclass_wf(x=x, y=y) print(result) ``` Key Points: - Hydra Initialization: Use Hydra's 
initialize
and 
compose
to load configurations. - Dataclass Handling: Ensure that your dataclasses are correctly defined and do not use 
Union
types directly, as Flyte's type system may not handle them well. Sources: - Flyte Dataclass Example - Flyte Issue on Union Types Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1723761661.728509 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
Open in Slack
PreviousNext
Powered by