Channels
#announcements
Title
m
Mike Ossareh
06/24/2022, 8:30 PMWe've a dataclass that has a parameter that is of type:
typing.List[typing.List[float]]looking through the test cases in flytekit i can see cases of List[List[int]] being serialized, but not float.
It seems
Transformer.get_transformercorrection: transformer.get_transformer(typing.List).to_python_value(...)
The serialized data looks a bit odd (ints are stored as floats when serialized).
Copy code
@dataclass_json
@dataclass
class Container:
floats: typing.List[typing.List[float]]
ints: typing.List[typing.List[int]]json_format.Parse() Copy code
collection {
literals {
scalar {
generic {
fields {
key: "floats"
value {
list_value {
values {
list_value {
values {
number_value: 1.2
}
}
}
values {
list_value {
values {
number_value: 4.7
}
}
}
}
}
}
fields {
key: "ints"
value {
list_value {
values {
list_value {
values {
number_value: 1.0
}
}
}
values {
list_value {
values {
number_value: 4.0
}
}
}
}
}
}
}
}
}
}but, ultimately it's clearly a List[List[]]
y
Yee
06/24/2022, 8:45 PM
what’s your test case?
let us add a test to flytekit - if we can repro we will address
m
Mike Ossareh
06/24/2022, 8:48 PM@Yee just putting it up on gist.github.com rn with some explaination
y
Yee
06/24/2022, 8:59 PM
let me take a look, will make a pr in a bit
thanks!
m
Mike Ossareh
06/27/2022, 3:06 PMgood morning; I wanted to check in on this. I'm happy to help work out what may be the root cause and submit a patch w/ tests; but there's enough about the inner workings of serialization that I'd at least need a pointer to some critical paths (i.e. clear test cases).
y
Yee
06/27/2022, 3:22 PM
yes… sorry, i will get to this asap
👍 1
can we chat about this tomorrow?
my short investigation is not really yielding results.
this is passing for me
Copy code
import typing
from dataclasses import dataclass
from dataclasses_json import dataclass_json
from google.protobuf import json_format
from marshmallow_jsonschema import JSONSchema
from flytekit.core.context_manager import FlyteContextManager
from flytekit.core.type_engine import (
TypeEngine,
convert_json_schema_to_python_class,
)
from flytekit.models.literals import Literal, LiteralCollection, Scalar
@dataclass_json
@dataclass()
class Container:
floats: typing.List[typing.List[float]]
ints: typing.List[typing.List[int]]
def test_nested_list_of_float2():
ctx = FlyteContextManager.current_context()
data = Container(floats=[[1.2, 2.4, 3.5], [4.7, 5.8, 6.0]], ints=[[1, 2, 3], [4, 5, 6]])
literal_type = TypeEngine.to_literal_type(Container)
lit = TypeEngine.to_literal(ctx, data, python_type=Container, expected=literal_type)
print(lit)
pv = TypeEngine.to_python_value(ctx, lit, Container)
print(pv)could you see if this passes for you too?
m
Mike Ossareh
06/28/2022, 3:48 PMI’ll give it a shot right now
I concur - this works. I guess my issue is in part not knowing how to use the inner parts of the API.
It’s not clear to me why our production code trips up on this then.
y
Yee
06/28/2022, 4:42 PM
hmm
can you explain again what you’re trying to accomplish in the production code?
as in, is there a way to just use the top level interface and avoid going deeper?
m
Mike Ossareh
06/28/2022, 6:05 PMI’m not entirely sure what is happening here; the code that was failing is now running without issue. This may have just been some transient issue with versions of code in one place mis-matching against code in another place.
Thanks for looking into it; if it turns out it crops up again I’ll do a better deep dive to understand what the issue is.
💯 1
2 Views