salmon-refrigerator-32115
11/15/2022, 12:14 AMtall-lock-23197
Resources
? You should request for resources in the task decorator: https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/customizing_resources.html.salmon-refrigerator-32115
11/15/2022, 5:49 PMsalmon-refrigerator-32115
11/15/2022, 5:50 PMtall-lock-23197
requests
as an argument: https://github.com/flyteorg/flytekit/blob/07299d0b0e8d172e33ad48d719f9014effb5d96b/flytekit/core/python_auto_container.py#L37.salmon-refrigerator-32115
11/15/2022, 5:56 PM@task
def get_xgb_task_resource_request(df: pd.DataFrame) -> Resources:
_Gibibyte = 1073741824
data_size_gb = df.memory_usage(index=True, deep=True).sum() // _Gibibyte
# request more memory for the task if the dataset size is >= 2 GB, because the default task mem alloc might not be enough.
resource_requests = Resources(mem=f'{data_size_gb}Gi') if data_size_gb >= 2 else None
return resource_requests
salmon-refrigerator-32115
11/15/2022, 5:57 PMsalmon-refrigerator-32115
11/15/2022, 5:59 PMsalmon-refrigerator-32115
11/15/2022, 7:18 PMtall-lock-23197
PythonInstanceTask
and requested resources in the execute()
method? Would you mind sharing the coding snippet?salmon-refrigerator-32115
11/16/2022, 5:18 PM_Gibibyte = 1073741824
data_size_gb = (train_size + validation_size) / _Gibibyte
"""
Dynamically request task memory in GB + 0.1 GB buffer for the task (to 3 decimals),
because the default task memory allocation might not be enough for big dataset
"""
self.resources.requests = Resources(mem=f'{round(data_size_gb + 0.1, 3)}Gi')
tall-lock-23197
salmon-refrigerator-32115
11/17/2022, 7:56 PMtall-lock-23197
salmon-refrigerator-32115
11/18/2022, 5:44 PMsalmon-refrigerator-32115
11/18/2022, 10:55 PMtall-lock-23197
salmon-refrigerator-32115
11/21/2022, 5:45 PMtall-lock-23197
tall-lock-23197
with_overrides
https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/customizing_resources.html#using-with-overrides?salmon-refrigerator-32115
11/22/2022, 5:45 AMtall-lock-23197
with_overrides()
should work with PythonInstanceTask class. Also regarding real-time resource calculation, I’m not sure how that can be done. I’ll defer to Kevin/Eduardo to answer.high-accountant-32689
11/22/2022, 6:49 PM@task
def t(a: int):
...
@dynamic
def dyn(a: int):
t(a=a).with_overrides(
requests=Resources(mem="432Mi"),
limits=Resources(mem="543Mi"),
)