https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
s

Sujith Samuel

08/25/2023, 4:36 PM
#ask-the-community, Hello All, I am trying to pass python dicts between tasks and i am getting the below error Failed to convert outputs of task 'train_model_task.prepare_training_data' at position 0: Object of type ndarray is not JSON serializable Can dicts not be passed between tasks, I am having no issues passing ndarrays and strings but this is causing issues now
l

Len Strnad

08/25/2023, 4:52 PM
What are the key and value types? Are you annotating the expected key and value types? As the message states, these types need to be json serializable.
s

Sujith Samuel

08/25/2023, 5:14 PM
Would it be please possible to share an example of such dicts being passed with keys and values defined in the function signatures
My keys are strings and the values are arrays
how can i define them in the task output section if the dict is having strings as keys and arrays as values
l

Len Strnad

08/25/2023, 5:19 PM
How big is your array?
If by array you mean numpy array, those are not json serializable as is from what I remember.
s

Sujith Samuel

08/25/2023, 5:20 PM
Out[18]:
Copy code
dict_values([array([[6.4, 2.8, 5.6, 2.2],
       [6.3, 2.7, 4.9, 1.8],
       [6.5, 3. , 5.8, 2.2],
       [5. , 3.2, 1.2, 0.2],
       [4.9, 3. , 1.4, 0.2],
       [7.6, 3. , 6.6, 2.1],
       [6.3, 2.8, 5.1, 1.5],
       [5.6, 3. , 4.5, 1.5],
       [6.9, 3.1, 5.4, 2.1],
       [6.7, 3. , 5.2, 2.3],
       [4.6, 3.1, 1.5, 0.2],
       [7.2, 3.6, 6.1, 2.5],
       [4.3, 3. , 1.1, 0.1],
       [5.4, 3. , 4.5, 1.5],
       [5.5, 2.5, 4. , 1.3],
       [7.9, 3.8, 6.4, 2. ],
       [7.2, 3.2, 6. , 1.8],
       [5.3, 3.7, 1.5, 0.2],
       [6.2, 2.8, 4.8, 1.8],
       [5.1, 3.4, 1.5, 0.2],
       [4.7, 3.2, 1.6, 0.2],
       [5.5, 2.4, 3.8, 1.1],
       [5.7, 4.4, 1.5, 0.4],
       [4.8, 3. , 1.4, 0.3],
       [6.2, 2.9, 4.3, 1.3],
       [5.6, 2.7, 4.2, 1.3],
       [5.7, 2.8, 4.5, 1.3],
       [5.5, 4.2, 1.4, 0.2],
       [6.7, 3. , 5. , 1.7],
       [5.9, 3.2, 4.8, 1.8],
       [6. , 2.2, 4. , 1. ],
       [5.1, 3.8, 1.5, 0.3],
       [5.8, 2.7, 3.9, 1.2],
       [4.9, 3.6, 1.4, 0.1],
       [6.8, 3.2, 5.9, 2.3],
       [6.3, 2.9, 5.6, 1.8],
       [5.8, 2.8, 5.1, 2.4],
       [7.3, 2.9, 6.3, 1.8],
       [5.5, 2.3, 4. , 1.3],
       [6.1, 2.8, 4. , 1.3],
       [5. , 3. , 1.6, 0.2],
       [6.5, 3. , 5.5, 1.8],
       [7.4, 2.8, 6.1, 1.9],
       [5.2, 3.5, 1.5, 0.2],
       [4.9, 3.1, 1.5, 0.1],
       [4.9, 2.4, 3.3, 1. ],
       [5.4, 3.4, 1.7, 0.2],
       [4.4, 2.9, 1.4, 0.2],
       [6.6, 3. , 4.4, 1.4],
       [6.3, 2.5, 5. , 1.9],
       [5. , 2. , 3.5, 1. ],
       [6.4, 2.7, 5.3, 1.9],
       [5.4, 3.9, 1.3, 0.4],
       [6.7, 3.3, 5.7, 2.1],
       [5.4, 3.4, 1.5, 0.4],
       [5. , 3.5, 1.3, 0.3],
       [7.2, 3. , 5.8, 1.6],
       [6.1, 3. , 4.9, 1.8],
       [6.9, 3.1, 5.1, 2.3],
       [7.1, 3. , 5.9, 2.1],
       [5.1, 3.5, 1.4, 0.2],
       [5.8, 2.7, 5.1, 1.9],
       [6.7, 3.1, 4.4, 1.4],
       [6.1, 2.6, 5.6, 1.4],
       [6.2, 2.2, 4.5, 1.5],
       [6.4, 3.1, 5.5, 1.8],
       [5.8, 2.7, 4.1, 1. ],
       [5.7, 2.5, 5. , 2. ],
       [7.7, 3.8, 6.7, 2.2],
       [5.5, 2.6, 4.4, 1.2],
       [6.3, 2.5, 4.9, 1.5],
       [6.1, 2.9, 4.7, 1.4],
       [5.8, 2.7, 5.1, 1.9],
       [6.7, 3.1, 5.6, 2.4],
       [6.1, 3. , 4.6, 1.4],
       [4.6, 3.4, 1.4, 0.3],
       [6. , 2.9, 4.5, 1.5],
       [4.9, 2.5, 4.5, 1.7],
       [4.7, 3.2, 1.3, 0.2],
       [5.1, 3.8, 1.6, 0.2],
       [5.4, 3.7, 1.5, 0.2],
       [5.1, 3.7, 1.5, 0.4],
       [5. , 3.3, 1.4, 0.2],
       [4.6, 3.2, 1.4, 0.2],
       [5.5, 3.5, 1.3, 0.2],
       [6.7, 3.1, 4.7, 1.5],
       [5.2, 2.7, 3.9, 1.4],
       [5.1, 2.5, 3. , 1.1],
       [6.9, 3.1, 4.9, 1.5],
       [6.1, 2.8, 4.7, 1.2],
       [6.3, 3.3, 4.7, 1.6],
       [5. , 3.6, 1.4, 0.2],
       [6.2, 3.4, 5.4, 2.3],
       [7.7, 2.8, 6.7, 2. ],
       [5.7, 3. , 4.2, 1.2],
       [6.4, 3.2, 5.3, 2.3],
       [5.7, 2.6, 3.5, 1. ],
       [5.9, 3. , 5.1, 1.8],
       [6.9, 3.2, 5.7, 2.3],
       [5.2, 4.1, 1.5, 0.1],
       [5.6, 2.9, 3.6, 1.3],
       [6.5, 3. , 5.2, 2. ],
       [4.8, 3.4, 1.6, 0.2],
       [5. , 3.5, 1.6, 0.6],
       [6.7, 2.5, 5.8, 1.8],
       [4.8, 3. , 1.4, 0.1],
       [6.3, 2.3, 4.4, 1.3],
       [6. , 3. , 4.8, 1.8],
       [5.6, 3. , 4.1, 1.3],
       [4.4, 3. , 1.3, 0.2],
       [6.3, 3.3, 6. , 2.5],
       [5.1, 3.8, 1.9, 0.4],
       [6.8, 3. , 5.5, 2.1],
       [4.6, 3.6, 1. , 0.2],
       [5.6, 2.5, 3.9, 1.1],
       [6.5, 2.8, 4.6, 1.5],
       [6.7, 3.3, 5.7, 2.5],
       [6.8, 2.8, 4.8, 1.4],
       [6.4, 3.2, 4.5, 1.5],
       [5. , 3.4, 1.5, 0.2]]), array([2, 2, 2, 0, 0, 2, 2, 1, 2, 2, 0, 2, 0, 1, 1, 2, 2, 0, 2, 0, 0, 1,
       0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 2, 2, 2, 2, 1, 1, 0, 2, 2, 0,
       0, 1, 0, 0, 1, 2, 1, 2, 0, 2, 0, 0, 2, 2, 2, 2, 0, 2, 1, 2, 1, 2,
       1, 2, 2, 1, 1, 1, 2, 2, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
       1, 1, 1, 0, 2, 2, 1, 2, 1, 2, 2, 0, 1, 2, 0, 0, 2, 0, 1, 2, 1, 0,
       2, 0, 2, 0, 1, 1, 2, 1, 1, 0])])
About a 100
This is the train_data for the iris dataset
l

Len Strnad

08/25/2023, 5:23 PM
Gotcha. Yeah, if each example is 1d, you could consider using pandas since that is automatically handled by flyte
In more complicated cases where each example is nd, I would probably write out to zarr and return a path to the zarr file possibly as a FlyteFile or FlyteDirectory or str. I write these files remotely unless I have FlyteFile upload it for me, but I would rather write once if it is sharded like zarr tends to be.
s

Sujith Samuel

08/25/2023, 5:27 PM
Thanks for your inputs. I will explore this
l

Len Strnad

08/25/2023, 5:29 PM
3 Views
Powered by