the new gant chart features is super awesome thanks for this

Question

the new gant chart features is super awesome thanks for this release it does however re raise a question as to why there is so much runtime overhead launching a workflow

Dylan Wilder · Answer

cc < Varun Kulkarni>

Dylan Wilder · Answer

we could make this a task but there are functional differences between tasks workflows so we d rather not

Dylan Wilder · Answer

also another question why does cache reading take 13s

Ketan (kumare3) · Answer

that sounds wrong

Ketan (kumare3) · Answer

it seems the deployment needs to be optimized

Ketan (kumare3) · Answer

we can help

Dylan Wilder · Answer

Something wrong with our propeller setup

Ketan (kumare3) · Answer

Ya seems like it

Dylan Wilder · Answer

update flagging this for the admins

Dylan Wilder · Answer

actually they re here slightly smiling face < Babis Kiosidis> thoughts on this

Babis Kiosidis · Answer

can take a look next week +1 thanks

Babis Kiosidis · Answer

is this cache latency affected by the `workflow reeval duration` config Which we intentionally set to 30s from default 10s to reduce the amount of checks during peak hours wdyt < Ketan kumare3 >

Ketan (kumare3) · Answer

I think the biggest thing would be max streaks

Ketan (kumare3) · Answer

We can safely make it 10 or even more

Babis Kiosidis · Answer

we set ```max streak length 10 workflow reeval duration 10s``` but both had 0 effect on these executions

Babis Kiosidis · Answer

Would it make sense to set a lower value at downstream eval the current value is 30s would 5s work or is there risk of side effects ```downstream eval duration 30s```

Ketan (kumare3) · Answer

< Babis Kiosidis> that is interesting Downstream eval is more important than workflow reval But it seems There is some other problem Are there too many old workflows hanging around Cc < Dan Rammer hamersaw >

Babis Kiosidis · Answer

about 4000 flyte workflow resources in our gke cluster currently which seems a bit elevated

Ketan (kumare3) · Answer

That is tiny

Ketan (kumare3) · Answer

Let me ping

Babis Kiosidis · Answer

we may expect around 2000+ at this time of the day i guess rough numbers

Ketan (kumare3) · Answer

I have not seen a problem till 50k+

Ketan (kumare3) · Answer

But mostly because of k8s throttling

Ketan (kumare3) · Answer

We can also use sharded propeller

Dylan Wilder · Answer

sounds fancy wink