Hi team, Is this expected? or are there workarou...
# ask-the-community
v
Hi team, Is this expected? or are there workarounds?
Copy code
flytekit.core.type_engine.RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently
k
Yes tuples are restricted
You can use list or dataclasses
n
this is a personal pain-point for me… also happens with NamedTuple
r
Aggh.. I just ran into this 😐
List of NamedTuples
v
i used dataclasses to avoid this @Rupsha Chaudhuri and they are cleaner than NamedTuples
r
Yeah.. I’m making the same switch now
k
I think we should support tuples in someway. The core team is thinking- community is welcome to propose and chime in
r
Curious why it’s “restricted”?
n
I believe it’s because we haven’t figured out a correct flyteidl/protobuf representation for tuples that’s completely compatible with python semantics:
Copy code
tuple[int, int]  # two integers
tuple[int, ...]  # any number of integers

# more generally
tuple[T1, T2]  # 2-tuple of types T1 and T2 respectively
tuple[T1, T2, ..., Tn]  # etc...
I think at the very least we should have better support for
NamedTuple
so that it can be arbitrarily nested.
k
Ya, we do have list support, so we could pass tuples as lists
r
I have a dataclass… one of the fields I want to pass around is a pandas dataframe. Unfortunately this results in the following error getting pegged as seen here
Copy code
f"Failed to extract schema for object {t}, (will run schemaless) error: {e}"
                f"If you have postponed annotations turned on (PEP 563) turn it off please. Postponed"
                f"evaluation doesn't work with json dataclasses"
Now 1 workaround would be to pass it around as a dictionary.. and then convert back to a dataframe but wondering if there’s a cleaner solution
k
hmm @Kevin Su dataclasses are supported right?
or is it only structured dataset
k
only structured dataset
because we can only use dataclass in the dataclass
structured dataset is a dataclass
k
cool, @Rupsha Chaudhuri can you use structured dataset?
r
Let me look into that
thanks
So I changed the pandas dataframes to list of dict… and now I’m seeing some weird errors when the workflow runs…
Copy code
We won't register PyTorchCheckpointTransformer, PyTorchTensorTransformer, and PyTorchModuleTransformer because torch is not installed.
Copy code
We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery
Not sure where these requirements are suddenly coming from
163 Views