Hi, I’m getting the following error (failed task with a dataclass/dataclass_json as an input and tuple of numpy arrays as an output) but I can’t seem to make sense of it. Looks like something to do with the task input, but any help with what might be causing this error would be appreciated:

Traceback (most recent call last):

      File "/opt/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
        native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 868, in literal_map_to_kwargs
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 868, in <dictcomp>
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 832, in to_python_value
        return transformer.to_python_value(ctx, lv, expected_python_type)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 606, in to_python_value
        return self._fix_dataclass_int(expected_python_type, self._deserialize_flyte_type(dc, expected_python_type))
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 589, in _fix_dataclass_int
        dc.__setattr__(f.name, self._fix_val_int(f.type, val))
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 568, in _fix_val_int
        ktype, vtype = DictTransformer.get_dict_types(t)

Message:

    not enough values to unpack (expected 2, got 0)

Would you mind sharing your code snippet?

Sure - this is the failing task

@task
def load_data(ds: SidetrekDataset) -> typing.Tuple[np.ndarray, np.ndarray]:
    # Load the dataset
    csv_data = load_dataset(ds, data_type="csv", compression="zip", streaming=False)

    df = pd.read_csv(csv_data)
    # df = pd.read_csv((pathlib.Path(__file__).parent / filename).resolve())
    # Define X and Y
    X = df.drop(["fraud"], axis=1)
    y = df["fraud"]
    # Standardize the data
    scaler = StandardScaler()
    X = scaler.fit_transform(X)
    return (X, y)

where

SidetrekDataset

is:

@dataclass_json
@dataclass
class SidetrekDataset(object):
    io: str
    source: str
    options: Dict

Can you specify

options

dict key and value types? Something like

Dict[str, int]

OK will try - thanks

It has to work but it isn't 😢 @high-accountant-32689 When I try to trigger the following workflow on the demo cluster:

@task
def t1(x: Dict[str, Any]) -> Dict[str, Any]:
    return x


@workflow
def wf(x: Dict[str, Any] = {"a": 1, "b": "hello"}):
    t1(x=x)

I see this error:

pyflyte run --remote test.py wf

{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
Failed with Exception: Reason: USER:AssertionError
Underlying Exception: The provided token has expired.
Failed to put data from /var/folders/6f/xcgm46ds59j7g__gfxmkgdf80000gn/T/flytevycut24g/control_plane_metadata/local_flytekit/e77e673b91c7b633f9931a10aeea228e to <s3://my-s3-bucket/data/f64272bf89185219386c7978c1df76d9/e77e673b91c7b633f9931a10aeea228e> (recursive=False).

Original exception: The provided token has expired.

Flytekit is trying to upload pickle file in the local environment.

@tall-lock-23197 , this is a separate issue, caused by how we handle default values for workflow arguments. What happens if you modify the invocation of

wf

like

pyflyte run --remote run test.py wf --x '{"a":1,"b":"hello"}'

?

I see

Failed to convert param <Option x>, {'a': 1, 'b': 'hello'} to typing.Dict[str, flytekit.types.pickle.pickle.FlytePickle.__class_getitem__.<locals>._SpecificFormatClass]

error.

@tall-lock-23197, https://github.com/flyteorg/flytekit/pull/1646 fixes this. Before we have a release, can you try running this with flytekit master?

Works for me!

Any idea as to when this will be released?