https://flyte.org logo
#ask-the-community
Title
# ask-the-community
e

Erik Dao

06/19/2023, 9:39 AM
[Errors with Flyte task optionally returning None] I have a simple Flyte task which fetches weather data from an API like this
Copy code
weather_data_op = NamedTuple(
    "WeatherDataOP",
    temperature=float,
    humidity=float,
    wind_speed=float
)

@task(cache=False, limits=Resources(cpu="1", mem="2Gi", ephemeral_storage="2Gi"))
def fetch_weather_data(latitude: float, longitude: float) -> weather_data_op:
    api_key = os.environ["WEATHER_API_KEY"]
    url = f"<http://api.openweathermap.org/data/2.5/weather?lat={latitude}&lon={longitude}&appid={api_key}>"
    res = requests.get(url)
    if not (res.status_code == 200):
        return None
    
    response = res.json()
    return weather_data_op(
        temperature=float(response['main']['temp']),
        humidity=float(response['main']['humidity']),
        wind_speed=float(response['wind']['speed']),
    )
Depending on the input longitude and latitude, the weather API might return None. When I execute this task for the inputs that return in None, Flyte will complain
Copy code
╭─────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────╮
│ in <module>:4                                                                                                   │
│                                                                                                                 │
│ ❱ 4 fetch_weather_data(latitude=1237.0, longitude=-154.0)                                                       │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/base_task.py:304 in         │
│ __call__                                                                                                        │
│                                                                                                                 │
│ ❱ 304 │   │   return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore                           │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/promise.py:1114 in          │
│ flyte_entity_call_handler                                                                                       │
│                                                                                                                 │
│ ❱ 1114 │   │   │   result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/base_task.py:285 in         │
│ local_execute                                                                                                   │
│                                                                                                                 │
│ ❱ 285 │   │   │   outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)                            │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/base_task.py:351 in         │
│ sandbox_execute                                                                                                 │
│                                                                                                                 │
│ ❱ 351 │   │   return self.dispatch_execute(ctx, input_literal_map)                                              │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/base_task.py:569 in         │
│ dispatch_execute                                                                                                │
│                                                                                                                 │
│ ❱ 569 │   │   │   │   │   expected_output_names[i]: native_outputs[i] for i, _ in enumerate(na                  │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'NoneType' object is not iterable
So I’m wondering what is a proper way to write a Flyte task that optionally returns None?
s

Samhita Alla

06/19/2023, 10:26 AM
I think
Optional[weather_data_op]
should work. Let me know if it isn't.
e

Erik Dao

06/19/2023, 10:43 AM
It’s not working for me. The error is
Copy code
RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently
s

Samhita Alla

06/19/2023, 10:59 AM
Looks like we need to support this. cc @Eduardo Apolinario (eapolinario)
k

Ketan (kumare3)

06/19/2023, 2:41 PM
What’s a tuple here
e

Erik Dao

06/19/2023, 3:49 PM
@Ketan (kumare3) I believe it’s the output when I used
-> Optional[weather_data_op]
or
-> Union[weather_data_op, None]
k

Ketan (kumare3)

06/19/2023, 4:57 PM
thats not a tuple
e

Erik Dao

06/19/2023, 5:11 PM
Indeed, it’s not a tuple. That’s why I’m confused about the error. FYI, here is the minimal code to reproduce the error. I can provide the weather api key via DM
Copy code
import os
import boto3
from typing import NamedTuple, Optional

from flytekit import task, workflow, Resources

os.environ["WEATHER_API_KEY"] = "xxxxxxx"

weather_data_op = NamedTuple(
    "WeatherDataOP",
    temperature=float,
    humidity=float,
    wind_speed=float
)


@task(cache=False, disable_deck=False, limits=Resources(cpu="1", mem="2Gi", ephemeral_storage="2Gi"))
def fetch_weather_data_task(latitude: float, longitude: float) -> Optional[weather_data_op]:
    api_key = os.environ["WEATHER_API_KEY"]
    url = f"<http://api.openweathermap.org/data/2.5/weather?lat={latitude}&lon={longitude}&appid={api_key}>"
    res = requests.get(url)
    if not (res.status_code == 200):
        None

    response = res.json()
    if 'main' not in response:
        return None

    return weather_data_op(
        temperature=response['main']['temp'],
        humidity=response['main']['humidity'],
        wind_speed=response['wind']['speed']
    )

@task
def simple_task() -> None:
    inputs = [
        {"latitude": 18.0, "longitude": -154.0},
        {"latitude": 1128.0, "longitude": -154.0}
    ]
    
    for item in inputs:
        weather_data = fetch_weather_data_task(latitude=item['latitude'], longitude=item['longitude'])
        if weather is not None:
            print(weather_data)

    
@workflow
def simple_workflow():
    simple_task()

    
if __name__ == "__main__":
    simple_workflow()
And here is the full stack trace
Copy code
Traceback (most recent call last) ───────────────────────────────────────╮
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:1203 in      │
│ get_literal_type                                                                                                │
│                                                                                                                 │
│ ❱ 1203 │   │   │   variants = [_add_tag_to_type(TypeEngine.to_literal_type(x), t.name) for (t,                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:1203 in      │
│ <listcomp>                                                                                                      │
│                                                                                                                 │
│ ❱ 1203 │   │   │   variants = [_add_tag_to_type(TypeEngine.to_literal_type(x), t.name) for (t,                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:824 in       │
│ to_literal_type                                                                                                 │
│                                                                                                                 │
│ ❱  824 │   │   res = transformer.get_literal_type(python_type)                                                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:213 in       │
│ get_literal_type                                                                                                │
│                                                                                                                 │
│ ❱  213 │   │   raise RestrictedTypeError(f"Transformer for type {self.python_type} is restricte                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently

During handling of the above exception, another exception occurred:

╭─────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────╮
│ in <module>:1                                                                                                   │
│                                                                                                                 │
│ ❱ 1 from flyte_demo.simple_workflow import simple_workflow                                                      │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/flyte_demo/simple_workflow.py:18 in <module>                                 │
│                                                                                                                 │
│ ❱ 18 def fetch_weather_data_task(latitude: float, longitude: float) -> Optional[weather_data_                   │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/task.py:259 in wrapper      │
│                                                                                                                 │
│ ❱ 259 │   │   task_instance = TaskPlugins.find_pythontask_plugin(type(task_config))(                            │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/tracker.py:35 in __call__   │
│                                                                                                                 │
│ ❱  35 │   │   o = super(InstanceTrackingMeta, cls).__call__(*args, **kwargs)                                    │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/python_function_task.py:121 │
│ in __init__                                                                                                     │
│                                                                                                                 │
│ ❱ 121 │   │   super().__init__(                                                                                 │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py:85 │
│ in __init__                                                                                                     │
│                                                                                                                 │
│ ❱  85 │   │   super().__init__(                                                                                 │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/base_task.py:420 in         │
│ __init__                                                                                                        │
│                                                                                                                 │
│ ❱ 420 │   │   │   interface=transform_interface_to_typed_interface(interface),                                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/interface.py:249 in         │
│ transform_interface_to_typed_interface                                                                          │
│                                                                                                                 │
│ ❱ 249 │   outputs_map = transform_variable_map(interface.outputs, output_descriptions)                          │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/interface.py:367 in         │
│ transform_variable_map                                                                                          │
│                                                                                                                 │
│ ❱ 367 │   │   │   res[k] = transform_type(v, descriptions.get(k, k))                                            │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/interface.py:386 in         │
│ transform_type                                                                                                  │
│                                                                                                                 │
│ ❱ 386 │   return _interface_models.Variable(type=TypeEngine.to_literal_type(x), description=de                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:824 in       │
│ to_literal_type                                                                                                 │
│                                                                                                                 │
│ ❱  824 │   │   res = transformer.get_literal_type(python_type)                                                  │
│                                                                                                                 │
│ /opt/bitnami/jupyterhub-singleuser/.local/lib/python3.8/site-packages/flytekit/core/type_engine.py:1206 in      │
│ get_literal_type                                                                                                │
│                                                                                                                 │
│ ❱ 1206 │   │   │   raise ValueError(f"Type of Generic Union type is not supported, {e}")                        │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Type of Generic Union type is not supported, Transformer for type <class 'tuple'> is restricted 
currently
s

Samhita Alla

06/20/2023, 4:19 AM
@Erik Dao, can you create an issue please?
e

Erik Dao

06/20/2023, 6:42 AM
@Samhita Alla Sure. Will try to do it today!
h

Heidi Hurst

06/22/2023, 10:14 AM
@Erik Dao Will you post the issue back in this thread once you create it? We're tracking a similar concern (cc: @Ailin Yu)
k

Ketan (kumare3)

06/22/2023, 1:18 PM
@Erik Dao / @Heidi Hurst / @Samhita Alla I found a bug in union transformer that should cause this. Fix in this PR
76 Views