I am developing a workflow that has two levels of ...
# flyte-support
q
I am developing a workflow that has two levels of fan out. I get a list[list[str]] and need to produce a list[list[FlyteDirectory]]. This is done by using two levels of map_task . Everything works fine locally but when executing remotely, I get this error:
Copy code
USER:AssertionError: error=Expected a directory, but the given uri '/tmp/flyte-9lq_c257/sandbox/local_flytekit/44d6ea3a6ffa4d930e874e79131e4c3fimsop046/tmphwg8139l' is not a directory.
The funny thing is that I don't get this error if I only have one level of map_task nor do I get this if I avoid using map_task and use two layers of dynamic tasks. Is this expected? I have a few observations: • The problem gets resolved if my innermost task returns a path String and lets Flyte create FlyteDirectory rather than constructing FlyteDirectory itself • The path pattens seem to be different when using one level of map_task and two levels ("/tmp/flytearog1v2o/control_plane_metadata/local_flytekit/..." vs "/tmp/flyte-9lq_c257/sandbox/local_flytekit/..." Here is the code for the inner most task
Copy code
@task(retries=0)  # type: ignore [misc]
def dummy_task(input: str) -> FlyteDirectory:
    working_dir = current_context().working_directory
    local_dir = Path(tempfile.mkdtemp(dir=working_dir))
    out_path = local_dir / "test.txt"
    abs_path = out_path.absolute()
    with abs_path.open("w") as f:
        f.write(input)

    # This fails remotely
    return FlyteDirectory(str(local_dir.absolute()))
    # But this succeeds!
    # return str(local_dir.absolute())
I'm trying to see if I am missing something obvious before filing a bug
t
can you file an issue please?
to clarify, we don’t support mapping over map tasks.
at least not yet
are you saying that that somehow works depending on the output type?
q
Yes
I didn't know that nested maps are not supported