After upgrading flytekit from 1.5.0 to 1.9.1, one ...
# flyte-supportn
numerous-hamburger-7178
09/08/2023, 6:11 PMAfter upgrading flytekit from 1.5.0 to 1.9.1, one of my workflows can't seem to serialize anymore. It invokes a map task. My other workflows that call map tasks will serialize properly. I can serialize the task by itself in the workflow just not when its being called with map_task. Any ideas? verbose stack trace in thread
numerous-hamburger-7178
09/08/2023, 6:12 PM Copy code
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ <venv>/lib/python3.10/site-packages/flytekit/core/tracker.py:18 in import_module_from_file │
│ │
│ ❱ 18 │ │ module = importlib.util.module_from_spec(spec) │
│ in module_from_spec:568 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'loader'
The above exception was the direct cause of the following exception:
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ <venv>/lib/python3.10/site-packages/flytekit/exceptions/scopes.py:199 in user_entry_point │
│ │
│ ❱ 199 │ │ │ │ return wrapped(*args, **kwargs) │
│ │
│ <workflow>.py:105 in workflow_name │
│ │
│ ❱ 105 │ map_task(task_name, concurrency=10)(info=[]) │
│ │
│ <venv>/lib/python3.10/site-packages/flytekit/core/map_task.py:338 in map_task │
│ │
│ ❱ 338 │ return MapPythonTask(task_function, concurrency=concurrency, min_success_ratio=min_s │
│ │
│ <venv>/lib/python3.10/site-packages/flytekit/core/tracker.py:78 in __call__ │
│ │
│ ❱ 78 │ │ mod_name, mod_file = InstanceTrackingMeta._find_instance_module() │
│ │
│ <venv>/lib/python3.10/site-packages/flytekit/core/tracker.py:69 in _find_instance_module │
│ │
│ ❱ 69 │ │ │ │ mod = InstanceTrackingMeta._get_module_from_main(frame.f_globals) │
│ │
│ <venv>/lib/python3.10/site-packages/flytekit/core/tracker.py:58 in _get_module_from_main │
│ │
│ ❱ 58 │ │ return import_module_from_file(module_name, file) │
│ │
│ <venv>/lib/python3.10/site-packages/flytekit/core/tracker.py:25 in import_module_from_file │
│ │
│ ❱ 25 │ │ raise ModuleNotFoundError(f"Module from file {file} cannot be loaded") from exc │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: Module from file <venv>/bin/pyflyte cannot be loadedg
glamorous-carpet-83516
09/08/2023, 6:22 PMCould you share a sample workflow to reproduce? that will be helpful
n
numerous-hamburger-7178
09/08/2023, 6:24 PM Copy code
@dataclass_json
@dataclass
class Info:
argx: str
@task(
requests=Resources(mem="8Gi", ephemeral_storage="16Gi", cpu="4"),
retries=0,
)
def task_y(info: Optional[Info]) -> None:
pass
@workflow
def workflow_r():
map_task(task_y)(info=[])gratitude thank you 1
numerous-hamburger-7178
09/08/2023, 6:25 PMits so baffling because I have other map tasks that are serializing preoprly
numerous-hamburger-7178
09/08/2023, 7:22 PMoh this is simpler but still erroring out for me
Copy code
from flytekit import map_task, task, workflow
@task
def task_y(info: str) -> str:
return info
@workflow
def workflow_r(inp: List[str]):
map_task(task_y)(info=inp)numerous-hamburger-7178
09/08/2023, 7:46 PMhttps://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.map_task.html#flytekit.map_task the example task here is also failing for me. 🤔
numerous-hamburger-7178
09/08/2023, 7:48 PMoh... it works if its being called inside a dynamic task
numerous-hamburger-7178
09/08/2023, 7:48 PM Copy code
@task
def my_mappable_task(a: int) -> Optional[str]:
return str(a)
@dynamic
def my_dynamic(x: List[int]):
return map_task(my_mappable_task, concurrency=10, min_success_ratio=0.75,)(
a=x
).with_overrides(cpu="10M")
@workflow
def my_wf2(x: List[int]):
my_dynamic(x=x)numerous-hamburger-7178
09/08/2023, 7:48 PMthe above works, but not
Copy code
@task
def my_mappable_task(a: int) -> typing.Optional[str]:
return str(a)
@workflow
def my_wf(x: typing.List[int]) -> typing.List[typing.Optional[str]]:
return map_task(my_mappable_task, metadata=TaskMetadata(retries=1), concurrency=10, min_success_ratio=0.75,)(
a=x
).with_overrides(cpu="10M")numerous-hamburger-7178
09/08/2023, 8:01 PMyea, I guess all my other workflows aside from this one have always used dynamic to call map_task. the one that was originally failing was the only one to directly call from a workflow
g
glamorous-carpet-83516
09/08/2023, 8:21 PM Copy code
@task
def my_mappable_task(a: int) -> typing.Optional[str]:
return str(a)
@workflow
def my_wf(x: typing.List[int]) -> typing.List[typing.Optional[str]]:
return map_task(my_mappable_task, metadata=TaskMetadata(retries=1), concurrency=10, min_success_ratio=0.75,)(
a=x
).with_overrides(cpu="10M")pyflyte runn
numerous-hamburger-7178
09/08/2023, 8:22 PMwith flytekit==1.9.1?
t
thankful-minister-83577
09/08/2023, 8:22 PM
also works for me.
g
glamorous-carpet-83516
09/08/2023, 8:22 PMyes, I used latest image. cr.flyte.org/flyteorg/flytekit:py3.10-latest
n
numerous-hamburger-7178
09/08/2023, 8:23 PM Copy code
pyflyte --verbose --pkgs foldername package --fast -p flytetester -d /foldername/src --image image -o $(git rev-parse HEAD).tgz --deref-symlinks -fnumerous-hamburger-7178
09/08/2023, 8:24 PMalso doesn't work with
Copy code
pyflyte register -p flytetester --domain development --destination-dir /foldername/src src/foldername -v ${TAG} --image imagenumerous-hamburger-7178
09/08/2023, 8:30 PM Copy code
pyflyte run src/folder/test_wf.py my_wf --x="[0,1]"g
glamorous-carpet-83516
09/08/2023, 8:34 PMit’s odd, could you try to create new python venv, and rerun it again.
n
numerous-hamburger-7178
09/08/2023, 10:23 PMrecreated env multiple times. it works for my teammate and I even took his pip freeze to create new envs but doesn't work for me for some reason 😕
2 Views