@Kevin Su In this workflow and in general, is it possible to add support for

dataclass

? We're looking into modifying flytekit/core/promise.py for example. I just wanted to confirm first if this functionality is even possible. Anotehr concern I have is what happens when a class contains another class as an attribute. It seems that not even NamedTuple supports this atm.

you mean support this?

@dataclass_json
@dataclass
class Person:
    name: str
    nationality: str

@task
def create_person(name: str, nationality: str) -> Person:
    return Person(name=name, nationality=nationality)

@task
def say_hello(name: str) -> str:
    greeting = f"hello {name}"
    print(greeting)
    return greeting

@workflow
def my_wf(name: str="aria", nationality:str="Indonesian") -> str:
    person = create_person(name=name, nationality=nationality)
    return say_hello(name=person.name). --> throws AttributeError: 'Promise' object has no attribute 'name'

yes exactly

yes, it’s possible. need to add a new field to the output_reference, like dataclass_var. propeller need to deserialize the upstream output, and extract specific attribute in the dataclass, and then pass it to downstream task. could you leave a comment about what you’re gonna change or one pager design in the issue. we can discuss in the issue. https://github.com/flyteorg/flyte/issues/3864

What if a dataclass contains another dataclass as an attribute? would this still work?

it should work as well. you should use nested class, like

message OutputReference {
    // Node id must exist at the graph layer.
    string node_id = 1;

    // Variable name must refer to an output variable for the node.
    string var = 2;

    Dataclass dataclass_var = 3;
}

message Dataclass {
    string var1 = 1;
    Dataclass var2 = 2;
}

@Richard Li @Byron Hsu can we talk about this later this week or early next?

this week better. richard’s internship ends next friday

in the meantime, which components should I be looking into? flyteidl, etc.?

i think the idl, kit, and, propeller These are my previous notes: 1. On the flytekit promise, add an

attribute

fields to the promise class. In the above case, promise

x

looks some like

Promise(NodeOutput=(t1, d), attribute".x")

2. On the flytepropeller side, in extractOutput, where the promise got resolved, access the attribute on the dataclass. dataclass is serialized as protobuf struct, so should be able to access the field by simply

.x

or something

i don’t think this will be done in a week though.

+ idl as kevin mentioned

this is for everything that’s json serializable right?

i don’t think this will be done in a week though.

the plan is just to get a design or poc

ah okay

getting done is a stretch

that’s possible yes

this is for everything that’s json serializable right?

yes

do you have time tomorrow?

yes

11am works for us, 11:30, 1:30, 2

works!

Will send zoom link at 11am I included the link in the email

i’ve forwarded to @Yee

I don’t think we need a full hour for this. But yeah let’s chat at 11 tomorrow