joe
03/09/2023, 11:21 PMYee
FLYTE_SDK_LOGGING_LEVEL=10
as an environment variable to the task and re-run please?Ketan (kumare3)
Dan Rammer (hamersaw)
03/10/2023, 3:59 PMPod
breakdown. The output of kubectl -n <namespace> get pod <pod-name> -o yaml
would be very helpful to breakdown pod scheduling, container pulling, etc. These are all very common performance issues.joe
03/10/2023, 5:58 PMDan Rammer (hamersaw)
03/10/2023, 6:14 PMstate:
terminated:
containerID: <docker://ab32398a800e951520d479878ccaca4a29782f4b116e5bdf17958cd2fb1bb67>b
exitCode: 0
finishedAt: "2023-03-10T17:50:56Z"
reason: Completed
startedAt: "2023-03-10T17:49:33Z"
This could be for a number of reasons, the first thing to do is increase the resource requests - currently it looks like 100m
CPU:
resources:
limits:
cpu: 100m
memory: 1Gi
requests:
cpu: 100m
memory: 1Gi
I'm assuming this is using fast register (ie. pyflyte run --remote ...
) which requires downloading and decompressing the code at runtime. This can be a little bit more resource intensive.Ketan (kumare3)
joe
03/11/2023, 12:13 AMcpu_request = "1000m"
@task(
limits=Resources(mem="1500Mi", cpu=cpu_request),
requests=Resources(cpu=cpu_request),
environment={"FLYTE_SDK_LOGGING_LEVEL": "10"},
)
def hello_world(name: str) -> str:
print(f"Hello {name}")
return name
Ketan (kumare3)
Dan Rammer (hamersaw)
03/11/2023, 1:17 AM