https://flyte.org
Join Slack
We've a dataclass that has a parameter that is of type: `typing.List[typing.List[float]]` ; it appea...
# announcements
f
We've a dataclass that has a parameter that is of type: 
typing.List[typing.List[float]]
; it appears to fail to serialize. Is this a known issue / limitation?
looking through the test cases in flytekit i can see cases of List[List[int]] being serialized, but not float.
It seems 
Transformer.get_transformer
is where the issue is cropping up; but I'm pretty unconvinced that my test case is correctly flexing this code path.
correction: transformer.get_transformer(typing.List).to_python_value(...)
The serialized data looks a bit odd (ints are stored as floats when serialized).
Copy code
@dataclass_json
@dataclass
class Container:
  floats: typing.List[typing.List[float]]
  ints: typing.List[typing.List[int]]
when passed through 
json_format.Parse()
and converted into literals it ends up looking like:
Copy code
collection {
  literals {
    scalar {
      generic {
        fields {
          key: "floats"
          value {
            list_value {
              values {
                list_value {
                  values {
                    number_value: 1.2
                  }
                }
              }
              values {
                list_value {
                  values {
                    number_value: 4.7
                  }
                }
              }
            }
          }
        }
        fields {
          key: "ints"
          value {
            list_value {
              values {
                list_value {
                  values {
                    number_value: 1.0
                  }
                }
              }
              values {
                list_value {
                  values {
                    number_value: 4.0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
but, ultimately it's clearly a List[List[]]
t
what’s your test case?
let us add a test to flytekit - if we can repro we will address
f
@thankful-minister-83577 just putting it up on gist.github.com rn with some explaination
t
let me take a look, will make a pr in a bit
thanks!
f
good morning; I wanted to check in on this. I'm happy to help work out what may be the root cause and submit a patch w/ tests; but there's enough about the inner workings of serialization that I'd at least need a pointer to some critical paths (i.e. clear test cases).
t
yes… sorry, i will get to this asap
👍 1
can we chat about this tomorrow?
my short investigation is not really yielding results.
this is passing for me
Copy code
import typing
from dataclasses import dataclass

from dataclasses_json import dataclass_json
from google.protobuf import json_format
from marshmallow_jsonschema import JSONSchema

from flytekit.core.context_manager import FlyteContextManager
from flytekit.core.type_engine import (
    TypeEngine,
    convert_json_schema_to_python_class,
)
from flytekit.models.literals import Literal, LiteralCollection, Scalar

@dataclass_json
@dataclass()
class Container:
    floats: typing.List[typing.List[float]]
    ints: typing.List[typing.List[int]]


def test_nested_list_of_float2():
    ctx = FlyteContextManager.current_context()
    data = Container(floats=[[1.2, 2.4, 3.5], [4.7, 5.8, 6.0]], ints=[[1, 2, 3], [4, 5, 6]])
    literal_type = TypeEngine.to_literal_type(Container)
    lit = TypeEngine.to_literal(ctx, data, python_type=Container, expected=literal_type)
    print(lit)

    pv = TypeEngine.to_python_value(ctx, lit, Container)
    print(pv)
could you see if this passes for you too?
f
I’ll give it a shot right now
I concur - this works. I guess my issue is in part not knowing how to use the inner parts of the API. It’s not clear to me why our production code trips up on this then.
t
hmm
can you explain again what you’re trying to accomplish in the production code?
as in, is there a way to just use the top level interface and avoid going deeper?
f
I’m not entirely sure what is happening here; the code that was failing is now running without issue. This may have just been some transient issue with versions of code in one place mis-matching against code in another place.
Thanks for looking into it; if it turns out it crops up again I’ll do a better deep dive to understand what the issue is.
💯 1
286 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by