https://flyte.org logo
#ask-the-community
Title
# ask-the-community
y

Yaniv Ben Zvi

03/26/2024, 9:20 AM
Hi all 👋 I'm trying to monitor a stale workflow - a workflow that is stuck in one of the following states: • There are not enough k8s resources for this workflow/tasks • Something is wrong with the workflow definition - bottlenecks, complex or poorly designed WF, etc. I need to catch up on the Flyte Metric. What is the best way to monitor this?
Also, I thought of using a timeout alert using the launch plan Slack alert.
r

Ryuu

03/26/2024, 9:27 AM
https://docs.flyte.org/en/latest/deployment/configuration/monitoring.html I have saw some Flyte metrics can be monitoring by garfana here
y

Yaniv Ben Zvi

03/26/2024, 9:29 AM
Yeah, I'm familiar with that; there are too many; I'm looking for metric suggestions
r

Ryuu

03/26/2024, 9:36 AM
Are you using Flyte binary @Yaniv Ben Zvi
y

Yaniv Ben Zvi

03/26/2024, 10:14 AM
No, I use the following chart: {“chartName”: “flyte”, “repoName”: “flyte”, “repoUrl”: “https://flyteorg.github.io/flyte”}
r

Ryuu

03/26/2024, 10:27 AM
@Yaniv Ben Zvi Here, there 3 flyte deployment path, If you deploy with Flyte chart, I think it's Flyte binary or Flyte Core now. About your question to customize the resources of K8s task Pod. Have to check 2 things: • Change Default limit Resources of Task Pod ( This could be config in the helm Chart ) • Your definded python task
Copy code
@task(requests=Resources(cpu="1", mem="100Mi"),  limits=Resources(cpu="2", mem="150Mi")) # The limits Resources's maximum == helm chart values config 
def count_unique_numbers(x: typing.List[int]) -> int:
    s = set()
    for i in x:
        s.add(i)
    return len(s)
@Yaniv Ben Zvi If you don't want to re-deploy your Flyte Core, You can update task-resource-attribute directly here
y

Yaniv Ben Zvi

03/26/2024, 11:17 AM
This is not helping me to monitor the Flyte stale WF.
r

Ryuu

03/26/2024, 11:40 AM
Oh so you want monitoring the wf by the metrics which contain info about resource using right ?
d

David Espejo (he/him)

03/26/2024, 12:57 PM
@Yaniv Ben Zvi I think for the conditions you describe, logging would me more helpful for you.
y

Yaniv Ben Zvi

03/26/2024, 1:12 PM
@Ryuu I want to know when there is some WF with execution duration above 2H, for example.
e

Eyal Abbas

03/26/2024, 4:46 PM
@Ryuu @David Espejo (he/him) what we are trying to do is to set a “timeout” warning at workflow level, similar to task timeout but only to alert on “long running wf executions”, ofc
wf_executions = sum(tasks_execution)
. for example, we have one wf with 100 tasks/nodes, we would like to schedule it multiple time during the day, what we don’t want is for one scheduled run to overlap another run, since Flyte doesn’t support this option, we want to at least get an alert if on extreme cases one run gets out of the normal duration to finish. the most convenient way would be to rely on prom metrics, we got lost there and we are not sure how to get to the right one. is there any metric (or combination of metrics) to give us the duration of a specific executions including tasks that are failed/finished or still running? if there are other solutions to the problem, they are also welcome 🙂
2 Views