Hi, I’m getting the following error (failed task w...
# ask-the-community
s
Hi, I’m getting the following error (failed task with a dataclass/dataclass_json as an input and tuple of numpy arrays as an output) but I can’t seem to make sense of it. Looks like something to do with the task input, but any help with what might be causing this error would be appreciated:
Copy code
Traceback (most recent call last):

      File "/opt/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
        native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 868, in literal_map_to_kwargs
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 868, in <dictcomp>
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 832, in to_python_value
        return transformer.to_python_value(ctx, lv, expected_python_type)
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 606, in to_python_value
        return self._fix_dataclass_int(expected_python_type, self._deserialize_flyte_type(dc, expected_python_type))
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 589, in _fix_dataclass_int
        dc.__setattr__(f.name, self._fix_val_int(f.type, val))
      File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 568, in _fix_val_int
        ktype, vtype = DictTransformer.get_dict_types(t)

Message:

    not enough values to unpack (expected 2, got 0)
s
Would you mind sharing your code snippet?
s
Sure - this is the failing task
Copy code
@task
def load_data(ds: SidetrekDataset) -> typing.Tuple[np.ndarray, np.ndarray]:
    # Load the dataset
    csv_data = load_dataset(ds, data_type="csv", compression="zip", streaming=False)

    df = pd.read_csv(csv_data)
    # df = pd.read_csv((pathlib.Path(__file__).parent / filename).resolve())
    # Define X and Y
    X = df.drop(["fraud"], axis=1)
    y = df["fraud"]
    # Standardize the data
    scaler = StandardScaler()
    X = scaler.fit_transform(X)
    return (X, y)
where
SidetrekDataset
is:
Copy code
@dataclass_json
@dataclass
class SidetrekDataset(object):
    io: str
    source: str
    options: Dict
s
Can you specify
options
dict key and value types? Something like
Dict[str, int]
s
OK will try - thanks
Is
Any
a valid type I can use? @Samhita Alla For example,
Dict[str, Any]
or will that not work? This dictionary should be able to receive many different types.
s
It has to work but it isn't 😢 @Eduardo Apolinario (eapolinario) When I try to trigger the following workflow on the demo cluster:
Copy code
@task
def t1(x: Dict[str, Any]) -> Dict[str, Any]:
    return x


@workflow
def wf(x: Dict[str, Any] = {"a": 1, "b": "hello"}):
    t1(x=x)
I see this error:
Copy code
pyflyte run --remote test.py wf

{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
{"asctime": "2023-05-17 09:47:13,339", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type typing.Any found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
Failed with Exception: Reason: USER:AssertionError
Underlying Exception: The provided token has expired.
Failed to put data from /var/folders/6f/xcgm46ds59j7g__gfxmkgdf80000gn/T/flytevycut24g/control_plane_metadata/local_flytekit/e77e673b91c7b633f9931a10aeea228e to <s3://my-s3-bucket/data/f64272bf89185219386c7978c1df76d9/e77e673b91c7b633f9931a10aeea228e> (recursive=False).

Original exception: The provided token has expired.
Flytekit is trying to upload pickle file in the local environment.
e
@Samhita Alla , this is a separate issue, caused by how we handle default values for workflow arguments. What happens if you modify the invocation of
wf
like
pyflyte run --remote run test.py wf --x '{"a":1,"b":"hello"}'
?
s
I see
Failed to convert param <Option x>, {'a': 1, 'b': 'hello'} to typing.Dict[str, flytekit.types.pickle.pickle.FlytePickle.__class_getitem__.<locals>._SpecificFormatClass]
error.
e
@Samhita Alla, https://github.com/flyteorg/flytekit/pull/1646 fixes this. Before we have a release, can you try running this with flytekit master?
s
Works for me!
s
Any idea as to when this will be released?
246 Views