https://flyte.org logo
#contribute
Title
# contribute
f

Fabio Grätz

01/30/2024, 2:55 PM
Copy code
@task(
    requests=Resources(
        cpu="3",
        mem="3Gi",
    ),
)
def foo():
    ...



@workflow
def my_wf():
    foo().with_overrides(
        requests=Resources(
            cpu="5",
            mem="5Gi",
        ),
    )
It just took me a while to figure out why this task pod ends up requesting 3 cpus/3Gi of memory instead of 5/5 which I would have expected. The reason is that the task is registered with limits=requests and then the override requests are larger than the original limits which is prevented here. I find this rather unintuitive. I think we should either automatically override the limits as well or log a warning that the user should override the limits too. Do you agree that this is unintuitive and if yes, which option do you think is better?
b

Bernhard Stadlbauer

01/30/2024, 4:41 PM
Would also find that unintuitive. Probably the harder bit is the other case of:
Copy code
@task(
    requests=Resources(
        cpu="3",
        mem="3Gi",
    ),
    limits=Resources(
        cpu="3",
        mem="3Gi",
    ),
)
def foo():
    ...



@workflow
def my_wf():
    foo().with_overrides(
        requests=Resources(
            cpu="5",
            mem="5Gi",
        ),
    )
As there’s probably no way to know whether
limits
have been set in the first place?
d

David Espejo (he/him)

01/30/2024, 5:07 PM
The reason is that the task is registered with limits=requests
Right, I think this is the case especially if no
limits
are specified. In any case, I'd + the idea of warning the user. I think it's a bit more deterministic at scale than automatically overriding the limits.
f

Fabio Grätz

01/30/2024, 9:04 PM
As there’s probably no way to know whether
limits
have been set in the first place?
If no limits have been set explicitly, the requests are taken also as limits.
https://github.com/flyteorg/flytekit/pull/2151 I still wonder whether this behaviour should be changed. I can’t imagine that any user would expect this result 👀
Does anyone know where limits are set to requests if not explicitly set on a task?
k

Ketan (kumare3)

01/30/2024, 9:46 PM
they are in propeller i think?
f

Fabio Grätz

01/30/2024, 10:04 PM
What do you think the behaviour should be here?
l

L godlike

01/31/2024, 2:35 AM
I think is on admin
I think is in this file
d

David Espejo (he/him)

01/31/2024, 4:43 PM
Added to tomorrow's contributors meetup agenda 👍🏽
2 Views