<#CP2HDHKE1|ask-the-community>, Hello All, I am tr...
# ask-the-community
s
#ask-the-community, Hello All, I am trying to pass python dicts between tasks and i am getting the below error Failed to convert outputs of task 'train_model_task.prepare_training_data' at position 0: Object of type ndarray is not JSON serializable Can dicts not be passed between tasks, I am having no issues passing ndarrays and strings but this is causing issues now
l
What are the key and value types? Are you annotating the expected key and value types? As the message states, these types need to be json serializable.
s
Would it be please possible to share an example of such dicts being passed with keys and values defined in the function signatures
My keys are strings and the values are arrays
how can i define them in the task output section if the dict is having strings as keys and arrays as values
l
How big is your array?
If by array you mean numpy array, those are not json serializable as is from what I remember.
s
Out[18]:
Copy code
dict_values([array([[6.4, 2.8, 5.6, 2.2],
       [6.3, 2.7, 4.9, 1.8],
       [6.5, 3. , 5.8, 2.2],
       [5. , 3.2, 1.2, 0.2],
       [4.9, 3. , 1.4, 0.2],
       [7.6, 3. , 6.6, 2.1],
       [6.3, 2.8, 5.1, 1.5],
       [5.6, 3. , 4.5, 1.5],
       [6.9, 3.1, 5.4, 2.1],
       [6.7, 3. , 5.2, 2.3],
       [4.6, 3.1, 1.5, 0.2],
       [7.2, 3.6, 6.1, 2.5],
       [4.3, 3. , 1.1, 0.1],
       [5.4, 3. , 4.5, 1.5],
       [5.5, 2.5, 4. , 1.3],
       [7.9, 3.8, 6.4, 2. ],
       [7.2, 3.2, 6. , 1.8],
       [5.3, 3.7, 1.5, 0.2],
       [6.2, 2.8, 4.8, 1.8],
       [5.1, 3.4, 1.5, 0.2],
       [4.7, 3.2, 1.6, 0.2],
       [5.5, 2.4, 3.8, 1.1],
       [5.7, 4.4, 1.5, 0.4],
       [4.8, 3. , 1.4, 0.3],
       [6.2, 2.9, 4.3, 1.3],
       [5.6, 2.7, 4.2, 1.3],
       [5.7, 2.8, 4.5, 1.3],
       [5.5, 4.2, 1.4, 0.2],
       [6.7, 3. , 5. , 1.7],
       [5.9, 3.2, 4.8, 1.8],
       [6. , 2.2, 4. , 1. ],
       [5.1, 3.8, 1.5, 0.3],
       [5.8, 2.7, 3.9, 1.2],
       [4.9, 3.6, 1.4, 0.1],
       [6.8, 3.2, 5.9, 2.3],
       [6.3, 2.9, 5.6, 1.8],
       [5.8, 2.8, 5.1, 2.4],
       [7.3, 2.9, 6.3, 1.8],
       [5.5, 2.3, 4. , 1.3],
       [6.1, 2.8, 4. , 1.3],
       [5. , 3. , 1.6, 0.2],
       [6.5, 3. , 5.5, 1.8],
       [7.4, 2.8, 6.1, 1.9],
       [5.2, 3.5, 1.5, 0.2],
       [4.9, 3.1, 1.5, 0.1],
       [4.9, 2.4, 3.3, 1. ],
       [5.4, 3.4, 1.7, 0.2],
       [4.4, 2.9, 1.4, 0.2],
       [6.6, 3. , 4.4, 1.4],
       [6.3, 2.5, 5. , 1.9],
       [5. , 2. , 3.5, 1. ],
       [6.4, 2.7, 5.3, 1.9],
       [5.4, 3.9, 1.3, 0.4],
       [6.7, 3.3, 5.7, 2.1],
       [5.4, 3.4, 1.5, 0.4],
       [5. , 3.5, 1.3, 0.3],
       [7.2, 3. , 5.8, 1.6],
       [6.1, 3. , 4.9, 1.8],
       [6.9, 3.1, 5.1, 2.3],
       [7.1, 3. , 5.9, 2.1],
       [5.1, 3.5, 1.4, 0.2],
       [5.8, 2.7, 5.1, 1.9],
       [6.7, 3.1, 4.4, 1.4],
       [6.1, 2.6, 5.6, 1.4],
       [6.2, 2.2, 4.5, 1.5],
       [6.4, 3.1, 5.5, 1.8],
       [5.8, 2.7, 4.1, 1. ],
       [5.7, 2.5, 5. , 2. ],
       [7.7, 3.8, 6.7, 2.2],
       [5.5, 2.6, 4.4, 1.2],
       [6.3, 2.5, 4.9, 1.5],
       [6.1, 2.9, 4.7, 1.4],
       [5.8, 2.7, 5.1, 1.9],
       [6.7, 3.1, 5.6, 2.4],
       [6.1, 3. , 4.6, 1.4],
       [4.6, 3.4, 1.4, 0.3],
       [6. , 2.9, 4.5, 1.5],
       [4.9, 2.5, 4.5, 1.7],
       [4.7, 3.2, 1.3, 0.2],
       [5.1, 3.8, 1.6, 0.2],
       [5.4, 3.7, 1.5, 0.2],
       [5.1, 3.7, 1.5, 0.4],
       [5. , 3.3, 1.4, 0.2],
       [4.6, 3.2, 1.4, 0.2],
       [5.5, 3.5, 1.3, 0.2],
       [6.7, 3.1, 4.7, 1.5],
       [5.2, 2.7, 3.9, 1.4],
       [5.1, 2.5, 3. , 1.1],
       [6.9, 3.1, 4.9, 1.5],
       [6.1, 2.8, 4.7, 1.2],
       [6.3, 3.3, 4.7, 1.6],
       [5. , 3.6, 1.4, 0.2],
       [6.2, 3.4, 5.4, 2.3],
       [7.7, 2.8, 6.7, 2. ],
       [5.7, 3. , 4.2, 1.2],
       [6.4, 3.2, 5.3, 2.3],
       [5.7, 2.6, 3.5, 1. ],
       [5.9, 3. , 5.1, 1.8],
       [6.9, 3.2, 5.7, 2.3],
       [5.2, 4.1, 1.5, 0.1],
       [5.6, 2.9, 3.6, 1.3],
       [6.5, 3. , 5.2, 2. ],
       [4.8, 3.4, 1.6, 0.2],
       [5. , 3.5, 1.6, 0.6],
       [6.7, 2.5, 5.8, 1.8],
       [4.8, 3. , 1.4, 0.1],
       [6.3, 2.3, 4.4, 1.3],
       [6. , 3. , 4.8, 1.8],
       [5.6, 3. , 4.1, 1.3],
       [4.4, 3. , 1.3, 0.2],
       [6.3, 3.3, 6. , 2.5],
       [5.1, 3.8, 1.9, 0.4],
       [6.8, 3. , 5.5, 2.1],
       [4.6, 3.6, 1. , 0.2],
       [5.6, 2.5, 3.9, 1.1],
       [6.5, 2.8, 4.6, 1.5],
       [6.7, 3.3, 5.7, 2.5],
       [6.8, 2.8, 4.8, 1.4],
       [6.4, 3.2, 4.5, 1.5],
       [5. , 3.4, 1.5, 0.2]]), array([2, 2, 2, 0, 0, 2, 2, 1, 2, 2, 0, 2, 0, 1, 1, 2, 2, 0, 2, 0, 0, 1,
       0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 2, 2, 2, 2, 1, 1, 0, 2, 2, 0,
       0, 1, 0, 0, 1, 2, 1, 2, 0, 2, 0, 0, 2, 2, 2, 2, 0, 2, 1, 2, 1, 2,
       1, 2, 2, 1, 1, 1, 2, 2, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
       1, 1, 1, 0, 2, 2, 1, 2, 1, 2, 2, 0, 1, 2, 0, 0, 2, 0, 1, 2, 1, 0,
       2, 0, 2, 0, 1, 1, 2, 1, 1, 0])])
About a 100
This is the train_data for the iris dataset
l
Gotcha. Yeah, if each example is 1d, you could consider using pandas since that is automatically handled by flyte
In more complicated cases where each example is nd, I would probably write out to zarr and return a path to the zarr file possibly as a FlyteFile or FlyteDirectory or str. I write these files remotely unless I have FlyteFile upload it for me, but I would rather write once if it is sharded like zarr tends to be.
s
Thanks for your inputs. I will explore this
l