https://flyte.org logo
Title
c

Calvin Leather

11/21/2022, 8:16 PM
Hey all, does anyone have experience using a dynamic with_overrides to set resource requests/limits at scale? We've been testing the following example in our dev cluster and its been working okay, wanted to start using this at scale and wanted to see if people are also doing this in larger workflows (e.g., 100's of dynamic nodes). I can't see any K8s or Flyte reason why this would have issues in the future (maybe funky interactions between heterogenous pod requests and K8s/EKS cluster autoscaling?), but its so nice and simple that it seems too good to be true.
@task
def test_task(input: int) -> int:
    import time
    time.sleep(600)
    return input + 1


@dynamic
def test_dynamic(list_of_ints: List[int]) -> List[int]:
    results = []
    for integer in list_of_ints:
        result = test_task(input=integer).with_overrides(requests=Resources(cpu=f"{integer}", mem=f"{integer}Gi"))
        results.append(result)
    return results
y

Yee

11/21/2022, 11:09 PM
does this run locally?
i’d be surprised if this does.
there’s a bug currently, dynamic task in local execution doesn’t use promises.
c

Calvin Leather

11/21/2022, 11:21 PM
Naw we just tested in our remote dev cluster
But thanks for that heads up
y

Yee

11/21/2022, 11:22 PM
yup. thanks for the unit test idea
k

Ketan (kumare3)

11/22/2022, 2:22 AM
@Calvin Leather this is ok and designed for specifically
It should scale drive