https://flyte.org
Join Slack
Is there an option to configure maximum parallelism (for map task)? it is set to 5 running tasks at ...
# flyte-support
e
Is there an option to configure maximum parallelism (for map task)? it is set to 5 running tasks at a time (by default I guess). I guess it is not related to the 
maxParallelism: 25
I configured in the flyteadmin configuration… Correct?
h
So maptasks have their own parallelism controls that defaults to executing all of the subtasks at the same time. This can be controlled using the 
concurrency
parameter - for example:
Copy code
mapped_out = map_task(
        a_mappable_task_2,
        metadata=TaskMetadata(retries=1),
        concurrency=2
    )(a=a)
🙏 1
🙏🏽 1
e
Thanks you! Do you know if there’s an option to configure it in the Helm chart (flyte-core)? cc: @important-hamburger-34837
đź‘€ 1
@hallowed-mouse-14616 We tried but it doesn’t work. Still the default parallelism for map tasks is 5. Which Flyte version do you use?
h
@elegant-toddler-67101 I just tested this with the lastest Flyte version and it is working correctly. My test code:
Copy code
@task(requests=Resources(cpu='2', mem='6Gi'))
def a_mappable_task_2(a: int) -> str:
    time.sleep(10)
    inc = a + 2
    stringified = str(inc)
    return stringified

@task
def coalesce(b: List[str]) -> str:
    coalesced = "".join(b)
    return coalesced

@workflow
def my_map_workflow_2(a: List[int]) -> str:
    mapped_out = map_task(
        a_mappable_task_2,
        metadata=TaskMetadata(retries=1),
        concurrency=2
    )(a=a)
    coalesced = coalesce(b=mapped_out)
    return coalesced
e
Thank you for the answer! This is exactly what we did. When we set the 
concurrency=12
, there were 5 running pods / 7 in queue… cc: @important-hamburger-34837
h
Are all 12 Pods created in k8s? but just 7 are not running?
e
Ok we got the issue! It’s the project memory quota that limits us for the maximum 5 tasks (= 5 running pods). When we set lower memory requests for the tasks - there were 10 tasks (= 10 pods) running at a time.
By the way, answering your question - it was 5 running tasks = 5 running pods, 7 tasks in queue = no pods are running. Only when pod is completed, another pod is being created. So, at a time we have 5 tasks only (based on the resources quota as I described in the comment above)
So, my question is: Is there any metric for the project resources quota/utilization? That way we can understand what is blocking from our map tasks to scale…
I checked on grafana explorer and couldn’t find any metric for that, neither in here https://docs.flyte.org/en/latest/deployment/configuration/monitoring.html
22 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by