New RunLLM response generated:
>>>*# Question*
So...
# runllm-flyte-triala
ancient-wolf-19325
08/16/2024, 2:37 PMNew RunLLM response generated:
>>>*# Question*
So I'm testing our
flyte-binaryv1.12.0sleep_more_minutespyflyte run --remote -p flytesnacks -d development testing.py sleep_more_minutes Copy code
from flytekit import workflow
from flytekit import task
import time
@task
def sleep_a_minute(seconds:int=60) -> int:
time.sleep(seconds)
return seconds
@workflow
def sleep_ten_minutes():
seconds = sleep_a_minute()
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
seconds = sleep_a_minute(seconds)
@workflow
def sleep_more_minutes():
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()
sleep_ten_minutes()ancient-wolf-19325
08/16/2024, 2:37 PM>># Answer Yes, the issue you're experiencing could be related to the memory usage problem described in this GitHub issue. The issue involves FlyteAdmin experiencing high memory usage due to repeated gRPC calls, which can lead to the pod being evicted by Kubernetes due to OOM (Out of Memory) errors. To track down the reason for the memory usage, consider the following steps: 1. Monitor Resource Usage: Continuously monitor the resource usage of FlyteAdmin and FlytePropeller. Use Kubernetes metrics and logs to identify any spikes in memory or CPU usage. 2. Check FlyteAdmin Configuration: Ensure that the FlyteAdmin configuration is optimized for your environment. Adjust the resource limits and liveness probe settings for FlyteAdmin. 3. Adjust Retry Mechanism: Tune down the retry mechanism introduced in the newer version to reduce the pressure on FlyteAdmin. 4. Review Authentication Setup: Ensure that the authentication setup between FlytePropeller and FlyteAdmin is correctly configured to avoid repeated authentication attempts. Would you like more detailed steps on how to implement these changes or further troubleshooting steps? Sources: - FlytePropeller 1.13.0 is crashing Flyteadmin Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1723819039.330009 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.