https://flyte.org logo
#ask-the-community
Title
# ask-the-community
r

Richard Li

08/23/2023, 5:54 PM
@Kevin Su In this workflow and in general, is it possible to add support for
dataclass
? We're looking into modifying flytekit/core/promise.py for example. I just wanted to confirm first if this functionality is even possible. Anotehr concern I have is what happens when a class contains another class as an attribute. It seems that not even NamedTuple supports this atm.
k

Kevin Su

08/23/2023, 6:07 PM
you mean support this?
Copy code
@dataclass_json
@dataclass
class Person:
    name: str
    nationality: str

@task
def create_person(name: str, nationality: str) -> Person:
    return Person(name=name, nationality=nationality)

@task
def say_hello(name: str) -> str:
    greeting = f"hello {name}"
    print(greeting)
    return greeting

@workflow
def my_wf(name: str="aria", nationality:str="Indonesian") -> str:
    person = create_person(name=name, nationality=nationality)
    return say_hello(name=person.name). --> throws AttributeError: 'Promise' object has no attribute 'name'
r

Richard Li

08/23/2023, 6:13 PM
yes exactly
k

Kevin Su

08/23/2023, 8:07 PM
yes, it’s possible. need to add a new field to the output_reference, like dataclass_var. propeller need to deserialize the upstream output, and extract specific attribute in the dataclass, and then pass it to downstream task. could you leave a comment about what you’re gonna change or one pager design in the issue. we can discuss in the issue. https://github.com/flyteorg/flyte/issues/3864
r

Richard Li

08/23/2023, 8:12 PM
What if a dataclass contains another dataclass as an attribute? would this still work?
k

Kevin Su

08/23/2023, 8:39 PM
it should work as well. you should use nested class, like
Copy code
message OutputReference {
    // Node id must exist at the graph layer.
    string node_id = 1;

    // Variable name must refer to an output variable for the node.
    string var = 2;

    Dataclass dataclass_var = 3;
}

message Dataclass {
    string var1 = 1;
    Dataclass var2 = 2;
}
y

Yee

08/23/2023, 9:20 PM
@Richard Li @Byron Hsu can we talk about this later this week or early next?
b

Byron Hsu

08/23/2023, 9:22 PM
this week better. richard’s internship ends next friday
r

Richard Li

08/23/2023, 9:25 PM
in the meantime, which components should I be looking into? flyteidl, etc.?
b

Byron Hsu

08/23/2023, 9:26 PM
i think the idl, kit, and, propeller These are my previous notes: 1. On the flytekit promise, add an
attribute
fields to the promise class. In the above case, promise
x
looks some like
Promise(NodeOutput=(t1, d), attribute".x")
2. On the flytepropeller side, in extractOutput, where the promise got resolved, access the attribute on the dataclass. dataclass is serialized as protobuf struct, so should be able to access the field by simply
.x
or something
y

Yee

08/23/2023, 9:26 PM
i don’t think this will be done in a week though.
b

Byron Hsu

08/23/2023, 9:26 PM
+ idl as kevin mentioned
y

Yee

08/23/2023, 9:26 PM
this is for everything that’s json serializable right?
b

Byron Hsu

08/23/2023, 9:26 PM
i don’t think this will be done in a week though.
the plan is just to get a design or poc
y

Yee

08/23/2023, 9:26 PM
ah okay
b

Byron Hsu

08/23/2023, 9:26 PM
getting done is a stretch
haha
y

Yee

08/23/2023, 9:27 PM
that’s possible yes
b

Byron Hsu

08/23/2023, 9:27 PM
this is for everything that’s json serializable right?
yes
y

Yee

08/23/2023, 9:27 PM
do you have time tomorrow?
b

Byron Hsu

08/23/2023, 9:27 PM
yes
y

Yee

08/23/2023, 9:29 PM
11am works for us, 11:30, 1:30, 2
b

Byron Hsu

08/23/2023, 9:54 PM
works!
@Richard Li can you send the invite?
r

Richard Li

08/23/2023, 10:02 PM
Will send zoom link at 11am I included the link in the email
b

Byron Hsu

08/23/2023, 10:03 PM
i’ve forwarded to @Yee
y

Yee

08/24/2023, 4:02 AM
I don’t think we need a full hour for this. But yeah let’s chat at 11 tomorrow
3 Views