https://flyte.org logo
m

Mike Ossareh

06/24/2022, 8:30 PM
We've a dataclass that has a parameter that is of type:
typing.List[typing.List[float]]
; it appears to fail to serialize. Is this a known issue / limitation?
looking through the test cases in flytekit i can see cases of List[List[int]] being serialized, but not float.
It seems
Transformer.get_transformer
is where the issue is cropping up; but I'm pretty unconvinced that my test case is correctly flexing this code path.
correction: transformer.get_transformer(typing.List).to_python_value(...)
The serialized data looks a bit odd (ints are stored as floats when serialized).
Copy code
@dataclass_json
@dataclass
class Container:
  floats: typing.List[typing.List[float]]
  ints: typing.List[typing.List[int]]
when passed through
json_format.Parse()
and converted into literals it ends up looking like:
Copy code
collection {
  literals {
    scalar {
      generic {
        fields {
          key: "floats"
          value {
            list_value {
              values {
                list_value {
                  values {
                    number_value: 1.2
                  }
                }
              }
              values {
                list_value {
                  values {
                    number_value: 4.7
                  }
                }
              }
            }
          }
        }
        fields {
          key: "ints"
          value {
            list_value {
              values {
                list_value {
                  values {
                    number_value: 1.0
                  }
                }
              }
              values {
                list_value {
                  values {
                    number_value: 4.0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
but, ultimately it's clearly a List[List[]]
y

Yee

06/24/2022, 8:45 PM
what’s your test case?
let us add a test to flytekit - if we can repro we will address
m

Mike Ossareh

06/24/2022, 8:48 PM
@Yee just putting it up on gist.github.com rn with some explaination
y

Yee

06/24/2022, 8:59 PM
let me take a look, will make a pr in a bit
thanks!
m

Mike Ossareh

06/27/2022, 3:06 PM
good morning; I wanted to check in on this. I'm happy to help work out what may be the root cause and submit a patch w/ tests; but there's enough about the inner workings of serialization that I'd at least need a pointer to some critical paths (i.e. clear test cases).
y

Yee

06/27/2022, 3:22 PM
yes… sorry, i will get to this asap
👍 1
can we chat about this tomorrow?
my short investigation is not really yielding results.
this is passing for me
Copy code
import typing
from dataclasses import dataclass

from dataclasses_json import dataclass_json
from google.protobuf import json_format
from marshmallow_jsonschema import JSONSchema

from flytekit.core.context_manager import FlyteContextManager
from flytekit.core.type_engine import (
    TypeEngine,
    convert_json_schema_to_python_class,
)
from flytekit.models.literals import Literal, LiteralCollection, Scalar

@dataclass_json
@dataclass()
class Container:
    floats: typing.List[typing.List[float]]
    ints: typing.List[typing.List[int]]


def test_nested_list_of_float2():
    ctx = FlyteContextManager.current_context()
    data = Container(floats=[[1.2, 2.4, 3.5], [4.7, 5.8, 6.0]], ints=[[1, 2, 3], [4, 5, 6]])
    literal_type = TypeEngine.to_literal_type(Container)
    lit = TypeEngine.to_literal(ctx, data, python_type=Container, expected=literal_type)
    print(lit)

    pv = TypeEngine.to_python_value(ctx, lit, Container)
    print(pv)
could you see if this passes for you too?
m

Mike Ossareh

06/28/2022, 3:48 PM
I’ll give it a shot right now
I concur - this works. I guess my issue is in part not knowing how to use the inner parts of the API. It’s not clear to me why our production code trips up on this then.
y

Yee

06/28/2022, 4:42 PM
hmm
can you explain again what you’re trying to accomplish in the production code?
as in, is there a way to just use the top level interface and avoid going deeper?
m

Mike Ossareh

06/28/2022, 6:05 PM
I’m not entirely sure what is happening here; the code that was failing is now running without issue. This may have just been some transient issue with versions of code in one place mis-matching against code in another place.
Thanks for looking into it; if it turns out it crops up again I’ll do a better deep dive to understand what the issue is.
💯 1
2 Views