side question from the above, a structured dataset...
# flytekitd
Dylan Wilder
08/01/2022, 3:45 PMside question from the above, a structured datasets and flyteschemas interoperable at all
y
Yee
08/01/2022, 3:48 PM
somewhat yes. for using cached outputs.
Yee
08/01/2022, 3:49 PM
basically if you have a pd.DataFrame output and another pd DF input in a subsequent task, but the first task was cached, yeah yes, the new structured dataset stuff can read it.
Yee
08/01/2022, 3:50 PM
but they’re not classes of each other in python, so you can’t write a task that outputs a FlyteSchema and another one that takes in a StructuredDataset
s
Samhita Alla
08/02/2022, 1:33 PM
@Yee, but FlyteSchema to StructuredDataset seems plausible according to https://github.com/flyteorg/flytesnacks/blob/master/cookbook/core/type_system/structured_dataset.py example.
If you see https://github.com/flyteorg/flytesnacks/blob/6997c290b9795752ae8d3a246c7ba96f69d20324/cookbook/core/type_system/structured_dataset.py#L247-L252, FlyteSchema is convertible to StructuredDataset.
160 Views