• JP Kosymna

    JP Kosymna

    5 months ago
    Hey all, I'm having trouble passing an empty list to a map_task. I think it might be a bug, hopefully there is a simple work around I could use for now. In the following example. Workflow
    test_fixed_array_map_task
    completes successfully in a few seconds, while workflow
    test_empty_array_map_task
    hangs indefinitely with the status QUEUED.
    from flytekit import map_task, task, workflow
    
    @task
    def say_hello(name: str) -> str:
        message = f"hello {name}"
        print(message)
        return message
    
    @workflow
    def test_empty_array_map_task() -> list[str]:
        names: list[str] = []
        messages = map_task(say_hello)(name=names)
        return messages
    
    @workflow
    def test_fixed_array_map_task() -> list[str]:
        names: list[str] = ['jp', 'ketan']
        messages = map_task(say_hello)(name=names)
        return messages
    Let me know if there is any additional information I can provide. Thanks
  • task config
    {
      "config": {},
      "id": {
        "resourceType": 1,
        "project": "flytesnacks",
        "domain": "development",
        "name": "flyte_workflows.example.mapper_say_hello_3",
        "version": "v60"
      },
      "type": "container_array",
      "metadata": {
        "runtime": {
          "type": 1,
          "version": "0.26.1",
          "flavor": "python"
        },
        "retries": {}
      },
      "interface": {
        "inputs": {
          "variables": {
            "name": {
              "type": {
                "collectionType": {
                  "simple": 3
                }
              },
              "description": "name"
            }
          }
        },
        "outputs": {
          "variables": {
            "o0": {
              "type": {
                "collectionType": {
                  "simple": 3
                }
              },
              "description": "o0"
            }
          }
        }
      },
      "taskTypeVersion": 1,
      "container": {
        "command": [],
        "args": [
          "pyflyte-map-execute",
          "--inputs",
          "{{.input}}",
          "--output-prefix",
          "{{.outputPrefix}}",
          "--raw-output-data-prefix",
          "{{.rawOutputDataPrefix}}",
          "--resolver",
          "flytekit.core.python_auto_container.default_task_resolver",
          "--",
          "task-module",
          "flyte_workflows.example",
          "task-name",
          "say_hello"
        ],
        "env": [
          {
            "key": "FLYTE_INTERNAL_IMAGE",
            "value": "<http://518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60|518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60>"
          }
        ],
        "config": [],
        "ports": [],
        "image": "<http://518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60|518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60>",
        "resources": {
          "requests": [],
          "limits": []
        }
      }
    }
  • Yee

    Yee

    5 months ago
    if you have easy access to propeller logs, would you mind doing a kubectl logs <propeller pod> | grep exec id?
  • JP Kosymna

    JP Kosymna

    5 months ago
    This line just repeats
    {
      "json": {
        "src": "controller.go:138"
      },
      "level": "info",
      "msg": "==> Enqueueing workflow [flytesnacks-development/l7mc2614ly]",
      "ts": "2022-02-17T19:33:52Z"
    }
  • Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    5 months ago
    Hey @JP Kosymna, thanks for reporting this. I think it's definitely a bug and have a few ideas where it could be. Do you mind filing an issue? I'll pick it up right away and pile it into all the other map task fixes I'm working on.
  • JP Kosymna

    JP Kosymna

    5 months ago
  • Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    5 months ago
    Thanks!
  • JP Kosymna

    JP Kosymna

    5 months ago
    This is blocking a job of mine. If you find any temporary fix please do post back here. Thanks
  • Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    @JP Kosymna can you use one element array instead?
  • JP Kosymna

    JP Kosymna

    5 months ago
    In my workflow I have a few tasks that produce lists of data to operate on later in the flow. I don't know that data ahead of time and in some cases it can be none. I guess I could return a nonsense value in case there is no data. The tasks I'm mapping over could then ignore that value.
  • Dan Rammer (hamersaw)

    Dan Rammer (hamersaw)

    5 months ago
    It looks like this is a specific check to ensure a map tasks size is > 0. I'll get working on a patch so that map tasks with no inputs transition immediately to success.
  • A nonsense value could certainly work, but obviously isn't a long-term solution. Once I have a PR it wouldn't be difficult to cut a propeller image from the patch and deploy that in your environment? I would be happy to help out with that. Otherwise, this will certainly be included in the next release.
  • JP Kosymna

    JP Kosymna

    5 months ago
    I already started the nonsense value solution so might stick with that. Is the next release on track for end of the month?
  • Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    @Dan Rammer (hamersaw) is working hard on it, we are hopeful all these issues to be addressed. Extremely sorry for the issues