https://flyte.org logo
#ask-the-community
Title
# ask-the-community
v

varsha Parthasarathy

11/03/2022, 7:10 PM
Hi team, Is this expected? or are there workarounds?
Copy code
flytekit.core.type_engine.RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently
k

Ketan (kumare3)

11/03/2022, 7:43 PM
Yes tuples are restricted
You can use list or dataclasses
n

Niels Bantilan

11/03/2022, 8:14 PM
this is a personal pain-point for me… also happens with NamedTuple
r

Rupsha Chaudhuri

12/16/2022, 9:35 PM
Aggh.. I just ran into this 😐
List of NamedTuples
v

varsha Parthasarathy

12/16/2022, 9:57 PM
i used dataclasses to avoid this @Rupsha Chaudhuri and they are cleaner than NamedTuples
r

Rupsha Chaudhuri

12/16/2022, 9:57 PM
Yeah.. I’m making the same switch now
k

Ketan (kumare3)

12/16/2022, 11:17 PM
I think we should support tuples in someway. The core team is thinking- community is welcome to propose and chime in
r

Rupsha Chaudhuri

12/16/2022, 11:18 PM
Curious why it’s “restricted”?
n

Niels Bantilan

12/19/2022, 3:12 PM
I believe it’s because we haven’t figured out a correct flyteidl/protobuf representation for tuples that’s completely compatible with python semantics:
Copy code
tuple[int, int]  # two integers
tuple[int, ...]  # any number of integers

# more generally
tuple[T1, T2]  # 2-tuple of types T1 and T2 respectively
tuple[T1, T2, ..., Tn]  # etc...
I think at the very least we should have better support for
NamedTuple
so that it can be arbitrarily nested.
k

Ketan (kumare3)

12/19/2022, 3:48 PM
Ya, we do have list support, so we could pass tuples as lists
r

Rupsha Chaudhuri

12/28/2022, 12:23 AM
I have a dataclass… one of the fields I want to pass around is a pandas dataframe. Unfortunately this results in the following error getting pegged as seen here
Copy code
f"Failed to extract schema for object {t}, (will run schemaless) error: {e}"
                f"If you have postponed annotations turned on (PEP 563) turn it off please. Postponed"
                f"evaluation doesn't work with json dataclasses"
Now 1 workaround would be to pass it around as a dictionary.. and then convert back to a dataframe but wondering if there’s a cleaner solution
k

Ketan (kumare3)

12/28/2022, 12:34 AM
hmm @Kevin Su dataclasses are supported right?
or is it only structured dataset
k

Kevin Su

12/28/2022, 12:56 AM
only structured dataset
because we can only use dataclass in the dataclass
structured dataset is a dataclass
k

Ketan (kumare3)

12/28/2022, 12:57 AM
cool, @Rupsha Chaudhuri can you use structured dataset?
r

Rupsha Chaudhuri

12/28/2022, 12:57 AM
Let me look into that
thanks
So I changed the pandas dataframes to list of dict… and now I’m seeing some weird errors when the workflow runs…
Copy code
We won't register PyTorchCheckpointTransformer, PyTorchTensorTransformer, and PyTorchModuleTransformer because torch is not installed.
Copy code
We won't register bigquery handler for structured dataset because we can't find the packages google-cloud-bigquery-storage and google-cloud-bigquery
Not sure where these requirements are suddenly coming from
8 Views