https://flyte.org logo
Join Slack
Powered by
Hey all, I'm having trouble passing an empty list ...
# announcements
j
Hey all, I'm having trouble passing an empty list to a map_task. I think it might be a bug, hopefully there is a simple work around I could use for now. In the following example. Workflow 
test_fixed_array_map_task
completes successfully in a few seconds, while workflow 
test_empty_array_map_task
hangs indefinitely with the status QUEUED.
Copy code
from flytekit import map_task, task, workflow

@task
def say_hello(name: str) -> str:
    message = f"hello {name}"
    print(message)
    return message

@workflow
def test_empty_array_map_task() -> list[str]:
    names: list[str] = []
    messages = map_task(say_hello)(name=names)
    return messages

@workflow
def test_fixed_array_map_task() -> list[str]:
    names: list[str] = ['jp', 'ketan']
    messages = map_task(say_hello)(name=names)
    return messages
Let me know if there is any additional information I can provide. Thanks
task config
Copy code
{
  "config": {},
  "id": {
    "resourceType": 1,
    "project": "flytesnacks",
    "domain": "development",
    "name": "flyte_workflows.example.mapper_say_hello_3",
    "version": "v60"
  },
  "type": "container_array",
  "metadata": {
    "runtime": {
      "type": 1,
      "version": "0.26.1",
      "flavor": "python"
    },
    "retries": {}
  },
  "interface": {
    "inputs": {
      "variables": {
        "name": {
          "type": {
            "collectionType": {
              "simple": 3
            }
          },
          "description": "name"
        }
      }
    },
    "outputs": {
      "variables": {
        "o0": {
          "type": {
            "collectionType": {
              "simple": 3
            }
          },
          "description": "o0"
        }
      }
    }
  },
  "taskTypeVersion": 1,
  "container": {
    "command": [],
    "args": [
      "pyflyte-map-execute",
      "--inputs",
      "{{.input}}",
      "--output-prefix",
      "{{.outputPrefix}}",
      "--raw-output-data-prefix",
      "{{.rawOutputDataPrefix}}",
      "--resolver",
      "flytekit.core.python_auto_container.default_task_resolver",
      "--",
      "task-module",
      "flyte_workflows.example",
      "task-name",
      "say_hello"
    ],
    "env": [
      {
        "key": "FLYTE_INTERNAL_IMAGE",
        "value": "<http://518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60|518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60>"
      }
    ],
    "config": [],
    "ports": [],
    "image": "<http://518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60|518673686532.dkr.ecr.us-west-2.amazonaws.com/flyte/test:v60>",
    "resources": {
      "requests": [],
      "limits": []
    }
  }
}
y
if you have easy access to propeller logs, would you mind doing a kubectl logs <propeller pod> | grep exec id?
j
This line just repeats
Copy code
{
  "json": {
    "src": "controller.go:138"
  },
  "level": "info",
  "msg": "==> Enqueueing workflow [flytesnacks-development/l7mc2614ly]",
  "ts": "2022-02-17T19:33:52Z"
}
d
Hey @JP Kosymna, thanks for reporting this. I think it's definitely a bug and have a few ideas where it could be. Do you mind filing an issue? I'll pick it up right away and pile it into all the other map task fixes I'm working on.
j
d
Thanks!
j
This is blocking a job of mine. If you find any temporary fix please do post back here. Thanks
k
@JP Kosymna can you use one element array instead?
j
In my workflow I have a few tasks that produce lists of data to operate on later in the flow. I don't know that data ahead of time and in some cases it can be none. I guess I could return a nonsense value in case there is no data. The tasks I'm mapping over could then ignore that value.
d
It looks like this is a specific check to ensure a map tasks size is > 0. I'll get working on a patch so that map tasks with no inputs transition immediately to success.
A nonsense value could certainly work, but obviously isn't a long-term solution. Once I have a PR it wouldn't be difficult to cut a propeller image from the patch and deploy that in your environment? I would be happy to help out with that. Otherwise, this will certainly be included in the next release.
j
I already started the nonsense value solution so might stick with that. Is the next release on track for end of the month?
k
@Dan Rammer (hamersaw) is working hard on it, we are hopeful all these issues to be addressed. Extremely sorry for the issues
177 Views
Open in Slack
PreviousNext
Powered by