jolly-florist-66728
09/26/2024, 6:43 PMwith_overrides() be used with map_task() ?
I often use e.g. some_python_task().with_overrides(...) to run python tasks with dynamically-computed resource quantities, and I'd like to do this for a mappable task that is called via map_task(), but it's not clear if this is supported.
# Answer
Yes, with_overrides() can be used with map_task(). According to the Flyte documentation, you can customize resource allocations for individual map tasks using with_overrides. Here's an example:
from flytekit import Resources, task, workflow, map_task
@task
def detect_anomalies(data_point: int) -> bool:
return data_point > 11
@workflow
def map_workflow_with_resource_overrides(data: list[int] = [10, 12, 11, 10, 13, 12, 100, 11, 12, 10]) -> list[bool]:
return map_task(detect_anomalies)(data_point=data).with_overrides(requests=Resources(mem="2Gi"))
However, there have been some issues reported with certain overrides not working as expected, such as container_image (GitHub Issue #4493).
Would you like more details on specific overrides or examples?
Sources:
- Flyte Documentation on Map Tasks
- GitHub Issue #4493
Slack Link
https://.slack.com/archives/CP2HDHKE1/p1727376170.635789 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.