honnix
09/13/2023, 3:31 PMmap_task
, it seems using .with_overrides
does not work. https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.map_task.html#flytekit-map-task gives the example overriding cpu but it doesn't seem to work. Directly setting that in @task
decorator works though. We are using flytekit 1.9.0.requests
and limits
.Dan Rammer (hamersaw)
09/13/2023, 5:44 PM@task
def my_mappable_task(a: int) -> typing.Optional[str]:
return str(a)
@workflow
def my_wf(x: typing.List[int]) -> typing.List[typing.Optional[str]]:
return map_task(my_mappable_task.with_overrides(cpu="10M"), metadata=TaskMetadata(retries=1), concurrency=10, min_success_ratio=0.75,)(
a=x
)
I think this will work - sorry don't have time to test right now.honnix
09/14/2023, 6:57 AMcpu="10M"
is no longer the correct way to set override.Dan Rammer (hamersaw)
09/14/2023, 12:37 PM@task
def a_mappable_task(a: int) -> str:
time.sleep(10)
inc = a + 2
stringified = str(inc)
return stringified
@task
def coalesce(b: List[str]) -> str:
coalesced = "".join(b)
return coalesced
@workflow
def my_map_workflow(a: List[int]) -> str:
mapped_out = map_task(a_mappable_task)(a=a).with_overrides(
requests=Resources(mem="300Mi"),
limits=Resources(mem="500Mi"),
retries=1,
)
coalesced = coalesce(b=mapped_out)
return coalesced
docs need to be updatedhonnix
09/14/2023, 2:34 PMDan Rammer (hamersaw)
09/14/2023, 3:52 PMhonnix
09/15/2023, 1:56 PM