Errors with Flyte task optionally returning None I have a si

Question

Errors with Flyte task optionally returning None I have a simple Flyte task which fetches weather data from an API like this ```weather data op = NamedTuple WeatherDataOP temperature=float humidity=float wind speed=float task cache=False limits=Resources cpu= 1 mem= 2Gi ephemeral storage= 2Gi def fetch weather data latitude float longitude float gt weather data op api key = os environ WEATHER API KEY url = f <http api openweathermap org data 2 5 weather lat= latitude amp lon= longitude amp appid= api key > res = requests get url if not res status code == 200 return None response = res json return weather data op temperature=float response main temp humidity=float response main humidity wind speed=float response wind speed ``` Depending on the input longitude and latitude the weather API might return None When I execute this task for the inputs that return in None Flyte will complain ``` Traceback most recent call last in lt module gt 4 4 fetch weather data latitude=1237 0 longitude= 154 0 opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core base task py 304 in call 304 return flyte entity call handler self args kwargs type ignore opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core promise py 1114 in flyte entity call handler 1114 result = cast LocallyExecutable entity local execute child ctx kwargs opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core base task py 285 in local execute 285 outputs literal map = self sandbox execute ctx input literal map opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core base task py 351 in sandbox execute 351 return self dispatch execute ctx input literal map opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core base task py 569 in dispatch execute 569 expected output names i native outputs i for i in enumerate na TypeError NoneType object is not iterable``` So I m wondering what is a proper way to write a Flyte task that optionally returns None

Samhita Alla · Answer

I think `Optional weather data op ` should work Let me know if it isn t

Erik Dao · Answer

It s not working for me The error is ```RestrictedTypeError Transformer for type lt class tuple gt is restricted currently```

Samhita Alla · Answer

Looks like we need to support this cc < Eduardo Apolinario eapolinario >

Ketan (kumare3) · Answer

What s a tuple here

Erik Dao · Answer

< Ketan kumare3 > I believe it s the output when I used ` gt Optional weather data op ` or ` gt Union weather data op None `

Ketan (kumare3) · Answer

thats not a tuple

Erik Dao · Answer

Indeed it s not a tuple That s why I m confused about the error FYI here is the minimal code to reproduce the error I can provide the weather api key via DM ```import os import boto3 from typing import NamedTuple Optional from flytekit import task workflow Resources os environ WEATHER API KEY = xxxxxxx weather data op = NamedTuple WeatherDataOP temperature=float humidity=float wind speed=float task cache=False disable deck=False limits=Resources cpu= 1 mem= 2Gi ephemeral storage= 2Gi def fetch weather data task latitude float longitude float gt Optional weather data op api key = os environ WEATHER API KEY url = f <http api openweathermap org data 2 5 weather lat= latitude amp lon= longitude amp appid= api key > res = requests get url if not res status code == 200 None response = res json if main not in response return None return weather data op temperature=response main temp humidity=response main humidity wind speed=response wind speed task def simple task gt None inputs = latitude 18 0 longitude 154 0 latitude 1128 0 longitude 154 0 for item in inputs weather data = fetch weather data task latitude=item latitude longitude=item longitude if weather is not None print weather data workflow def simple workflow simple task if name == main simple workflow ``` And here is the full stack trace ```Traceback most recent call last opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 1203 in get literal type 1203 variants = add tag to type TypeEngine to literal type x t name for t opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 1203 in lt listcomp gt 1203 variants = add tag to type TypeEngine to literal type x t name for t opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 824 in to literal type 824 res = transformer get literal type python type opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 213 in get literal type 213 raise RestrictedTypeError f Transformer for type self python type is restricte RestrictedTypeError Transformer for type lt class tuple gt is restricted currently During handling of the above exception another exception occurred Traceback most recent call last in lt module gt 1 1 from flyte demo simple workflow import simple workflow opt bitnami jupyterhub singleuser flyte demo simple workflow py 18 in lt module gt 18 def fetch weather data task latitude float longitude float gt Optional weather data opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core task py 259 in wrapper 259 task instance = TaskPlugins find pythontask plugin type task config opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core tracker py 35 in call 35 o = super InstanceTrackingMeta cls call args kwargs opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core python function task py 121 in init 121 super init opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core python auto container py 85 in init 85 super init opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core base task py 420 in init 420 interface=transform interface to typed interface interface opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core interface py 249 in transform interface to typed interface 249 outputs map = transform variable map interface outputs output descriptions opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core interface py 367 in transform variable map 367 res k = transform type v descriptions get k k opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core interface py 386 in transform type 386 return interface models Variable type=TypeEngine to literal type x description=de opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 824 in to literal type 824 res = transformer get literal type python type opt bitnami jupyterhub singleuser local lib python3 8 site packages flytekit core type engine py 1206 in get literal type 1206 raise ValueError f Type of Generic Union type is not supported e ValueError Type of Generic Union type is not supported Transformer for type lt class tuple gt is restricted currently```

Samhita Alla · Answer

< Erik Dao> can you create an issue please

Erik Dao · Answer

< Samhita Alla> Sure Will try to do it today

Heidi Hurst · Answer

< Erik Dao> Will you post the issue back in this thread once you create it We re tracking a similar concern cc < Ailin Yu>

Ketan (kumare3) · Answer

< Erik Dao> < Heidi Hurst> < Samhita Alla> I found a bug in union transformer that should cause this Fix in this PR