https://flyte.org logo
Join Slack
Powered by
<@U0265RTUJ5B> <@UNR3C6Y4T> <@USU6W5ATA> Is it pos...
# hacktoberfest-2022
t
@high-accountant-32689 @thankful-minister-83577 @glamorous-carpet-83516 Is it possible to use 
LiteralCollection
with multiple literal types? If so, how should the 
get_literal_type
method be defined since 
collection_type
can only accept a single 
LiteralType
? See https://github.com/flyteorg/flytekit/pull/1269/files PR for reference.
f
It is possible but not recommend- the underlying type system allows multi variate lists, but it has been clamped down as static type assertions are not possible Also don’t know a usecase
t
I want to return two literals — a blob and a string — from the 
to_literal
method. What’s the preferred way to implement this?
f
Why?
t
Blob contains a serialized tensor and string is the datatype of the tensor.
f
Why do t you add the te sir type in metadata
Like we do in json schema
Or in format?
Happy to discuss
t
Um, format makes sense but it usually pertains to the file type, right? But I guess it’s okay to send the dtype of the tensor.
Cool. Thanks Ketan! I’ll make the required change.
b
When I'm trying to apply the changes you'd suggested me, I'm facing few errors
Copy code
@pytest.mark.parametrize(
        "transformer, python_val",
        [
            (TensorFlowTensorTransformer(),  result_tensor),
        ],
    )
    def test_get_literal_type(transformer, python_val):
        tf = transformer
        lt = tf.get_literal_type(python_val)
>       assert lt == LiteralType(blob=BlobType(format=python_val.dtype.name, dimensionality=BlobType.BlobDimensionality.SINGLE))
E       assert blob {\n  format: "TensorflowTensor"\n}\n == blob {\n  format: "float32"\n}\n
E        +  where blob {\n  format: "float32"\n}\n = LiteralType(blob=format: "float32"\n)
E        +    where format: "float32"\n = BlobType(format='float32', dimensionality=0)
E        +      where 'float32' = tf.float32.name
E        +        where tf.float32 = <tf.Tensor: shape=(2, 2), dtype=float32, numpy=\narray([[22., 28.],\n       [49., 64.]], dtype=float32)>.dtype
E        +      and   0 = <class 'flytekit.models.core.types.BlobType.BlobDimensionality'>.SINGLE
E        +        where <class 'flytekit.models.core.types.BlobType.BlobDimensionality'> = BlobType.BlobDimensionality

tests/flytekit/unit/extras/tensorflow/test_transformations.py:36: AssertionError
______________ test_to_python_value_and_literal[transformer0-float32-python_val0] ______________
Why am I seeing "TensorflowTensor" in this line?
Copy code
assert blob {\n  format: "TensorflowTensor"\n}\n == blob {\n  format: "float32"\n}\n
@tall-lock-23197
I've changed format to be 
python_val.dtype.name
t
Have you modified the test to reflect the dtype?
b
I tried narrowing down, and it seems like i'm running a cached buggy version of my code, pytest too seems to run a cached version. I'll try creating a fresh virtual env and see what happens.
@tall-lock-23197 It turns out to be an issue with my virtual env. I've took the pains to set up a fresh env and everything works fine now. There are couple of errors with one of my tests i'll be soon fixing them and pushing the code.
t
Awesome! Thanks, @billowy-continent-33554! Appreciate all the work. 🙂
b
@tall-lock-23197 Uh oh.. there's a conflict here when I specify 
format=python_val.dtype.name
, test_tensor.py raises an error (the other test passes)
Copy code
format=python_val.dtype.name,
E   AttributeError: 'property' object has no attribute 'name'
and when I only used 
format=python_val.dtype
test_transformations.py raises an error that it cannot accept anything other than unicode/bytes (test_tensor.py passes)
Copy code
>       return _types_pb2.BlobType(format=self.format, dimensionality=self.dimensionality)
E       TypeError: tf.float32 has type DType, but expected one of: bytes, unicode

flytekit/models/core/types.py:69: TypeError
I tried converting the type object to string like : 
str(python_val.dtype)
to resolve the errors. Is it a valid approach?
t
Um, I guess no cause 
dtype.name
should exist.
b
I don't why it says property has no attribute name
More verbose error code:
Copy code
tests/flytekit/unit/extras/tensorflow/test_tensor.py:8: in <module>
    def create_rank1_tensor() -> tf.Tensor:
flytekit/core/task.py:212: in task
    return wrapper(_task_function)
flytekit/core/task.py:195: in wrapper
    task_instance = TaskPlugins.find_pythontask_plugin(type(task_config))(
flytekit/core/tracker.py:35: in __call__
    o = super(InstanceTrackingMeta, cls).__call__(*args, **kwargs)
flytekit/core/python_function_task.py:121: in __init__
    super().__init__(
flytekit/core/python_auto_container.py:74: in __init__
    super().__init__(
flytekit/core/base_task.py:386: in __init__
    interface=transform_interface_to_typed_interface(interface),
flytekit/core/interface.py:220: in transform_interface_to_typed_interface
    outputs_map = transform_variable_map(interface.outputs, output_descriptions)
flytekit/core/interface.py:328: in transform_variable_map
    res[k] = transform_type(v, descriptions.get(k, k))
flytekit/core/interface.py:346: in transform_type
    return _interface_models.Variable(type=TypeEngine.to_literal_type(x), description=description)
flytekit/core/type_engine.py:710: in to_literal_type
    res = transformer.get_literal_type(python_type)
flytekit/extras/tensorflow/tensor.py:26: in get_literal_type
    format=python_val.dtype.name,
E   AttributeError: 'property' object has no attribute 'name'
t
Can you share with me the 
get_literal_type
method?
b
Copy code
def get_literal_type(self, python_val: tf.Tensor) -> LiteralType:
        return LiteralType(
            blob=_core_types.BlobType(
                format=python_val.dtype.name,
                dimensionality=_core_types.BlobType.BlobDimensionality.SINGLE,
            )
        )
How about 
str(python_val.dtype.name)
?
t
get_literal_type
doesn’t accept 
python_val
. It needs to be 
t
that corresponds to the type info.
I’m not sure if we can extract type info from 
t
to initialize 
format
. Will need to check.
b
It seems we need one more function to do that
I’m not sure if we can extract type info from 
t
to initialize 
format
. Will need to check.
I don't think so. It is something that's associated with the items of 
python_val
or tensor object
t
Can you fix the rest and I’ll see if there’s a way to fill that out in the get_literal_type method?
b
okay, i'll see
154 Views
Open in Slack
PreviousNext
Powered by