Jimmy Du
06/28/2023, 1:13 AM./bin/pyflyte run --remote core/flyte_basics/hello_world.py my_wf
an allocation of 300MiB results in OOM, 500MiB allowed workflow to complete successfully).
Running hello_world.py without importing flyte modules seems to use 0.55MiB.
Are there potential settings I could look into to optimize memory usage? Looking into using Flyte for long running operations, so reducing memory overhead as much as possible would help reduce costs for these long running concurrent workflows.Ketan (kumare3)
Jimmy Du
06/29/2023, 12:32 AMFilename: core/flyte_basics/hello_world.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
28 142.3 MiB 142.3 MiB 1 @workflow
29 @profile
30 def my_wf() -> str:
31 142.3 MiB 0.0 MiB 1 res = say_hello()
32 142.3 MiB 0.0 MiB 1 return res
Filename: core/flyte_basics/hello_world.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
28 142.4 MiB 142.4 MiB 1 @workflow
29 @profile
30 def my_wf() -> str:
31 142.4 MiB 0.1 MiB 1 res = say_hello()
32 142.4 MiB 0.0 MiB 1 return res
Profile for hello_world.py without workflow/task annotations:
Filename: hello_world_test.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
25 19.0 MiB 19.0 MiB 1 @profile
26 def my_wf() -> str:
27 19.0 MiB 0.0 MiB 1 res = say_hello()
28 19.0 MiB 0.0 MiB 1 return res
Ketan (kumare3)