https://flyte.org
Join Slack
the new gant chart features is super awesome! thanks for this release. it does however re-raise a q...
# flytekit
f
the new gant chart features is super awesome! thanks for this release. it does however re-raise a question as to why there is so much runtime overhead launching a workflow
cc @lively-sundown-82704
we could make this a task, but there are functional differences between tasks/workflows so we'd rather not
also, another question: why does cache reading take 13s?
f
that sounds wrong
it seems the deployment needs to be optimized
we can help
f
Something wrong with our propeller setup?
f
Ya seems like it
f
update: flagging this for the admins
actually, they're here 🙂 @swift-animal-75798 thoughts on this?
s
can take a look next week 👍 thanks
is this cache latency affected by the 
workflow-reeval-duration
config? Which we intentionally set to 30s (from default 10s) to reduce the amount of checks during peak hours? wdyt @freezing-airport-6809?
f
I think the biggest thing would be - max-streaks
We can safely make it 10 or even more
s
we set:
Copy code
max-streak-length: 10
workflow-reeval-duration: 10s
but both had 0 effect on these executions
Would it make sense to set a lower value at downstream eval? the current value is 30s, would 5s work or is there risk of side-effects ?
Copy code
downstream-eval-duration: 30s
👍 1
f
@swift-animal-75798 that is interesting. Downstream eval is more important than workflow reval. But it seems There is some other problem. Are there too many old workflows hanging around? Cc @hallowed-mouse-14616
s
about 4000 flyte workflow resources in our gke cluster currently, which seems a bit elevated
f
That is tiny
Let me ping
s
we may expect around 2000+- at this time of the day i guess. (rough numbers)
f
I have not seen a problem till 50k+
But, mostly because of k8s throttling
We can also use sharded propeller
f
sounds fancy 😉
162 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by