https://flyte.org logo
Join Slack
Powered by
I have a question about the expected flytepropelle...
# flyte-v1-support
n
I have a question about the expected flytepropeller performance with a large number of pods in combination with array node/map task: • Workflow with 4 to 5 map tasks, between 5 to 15k pods existing at the same time. • I'm seeing that propeller sometimes doesn't look at the status of some completed pods for hours (have seen up to 10h) ◦ (I put print statements into plugin manager to see in which phase which resource is evaluated, they are not evaluated despite having completed so this is not related to errors sending update events to admin) • Sometimes the succeeded pods have been garbage collected and propeller treats the "missing" pod as a failure I'm aware that I can prevent the last point with 
inject-finalizer
to at least get eventual consistency. But my question is whether propeller not evaluating pods for hours in such a scenario is expected or unexpected. I know that I can shard propeller but this would only help me if I break this down into multiple workflows? Any other parameters I can tune so have propeller evaluate the pods earlier? Thank you!
👀 1
a
We've typically used a combination of the 
max-ttl-hours
and 
gc-interval
to control this
n
GCInterval config.Duration 
json:"gc-interval" pflag:"Run periodic GC every 30 minutes"
MaxTTLInHours int 
json:"max-ttl-hours" pflag:"Maximum number of hours a completed workflow should be retained. Number between 1-23 hours"
@alert-oil-1341 thank you for the quick reply. Could you please elaborate how these parameters can help in this scenario? In my scenario the workflow is still running but the status in admin of some pods has not been updated for hours.
a
Ah, yea now I get what you're talking about. No, I don't think I've seen any configuration directly for this case. Curious, the missing pod failures, where does that propagate? In logs somewhere? I'm curious to go check my logs to see if it's something we're seeing.
n
I see it in GCP audit logs. • garbage collector deletes pods • flytepropeller tries to delete the pod but its already gone, treated as error
But is not seeing updates for hours something you see too at this scale of number of pods?
Of is this unexpected for you?
Trying to look through all parameters I could tune: 
downstream-eval-duration: 30s
does this mean that we re evaluate all pods, rayjobs, ... every 30 seconds by default? Is my understanding correct that this is only a fallback mechanism and that by default the watch api should push on changes? Could it be that re checking every 30s becomes an issue when running >10k pods, i.e. that it might help increasing this?
publish-k8s-events
does anyone know whether disabling this helps?
a
was also looking into 
kube-client-config
I've been poking around while doing other things, but to answer your question - no, we don't have quite that volume. We have set the 
downstream-eval-duration
lower and sharded, but we've also been having issues w/ propeller crashes, so I'm more interested in the outcome. I don't think we've got a solution
n
downstream-eval-duration
lower as in less than 30s? Isn't this the interval after which the informer cache is completely resynched? My current thinking is that this should be higher at high load, e.g. a few minutes to reduce stress on the api server. Isn't this complete resync just a fallback mechanism in case watch events got lost?
@high-park-82026 can I please ask you whether you know whether 
downstream-eval-duration: 30s
can lead to issues when running >10k pods as in "resynching the informer cache too often"? (Sorry for pinging here)
a
Yea, as I said, lower could be completely wrong
We're still on the hunt for optimal values everywhere
re: 
kube-client-config
Copy code
The Kube client config controls the request throughput from FlytePropeller to the Kube API server. These requests may include creating/monitoring Pods or creating/updating FlyteWorkflow CRDs to track workflow execution. The default configuration (provided by k8s) contains very steep rate-limiting, and therefore FlytePropeller provides a default configuration that offers better performance. However, if your workload involves larger scales (e.g., >5k fanout dynamic or map tasks, >8k concurrent workflows, etc.,) the Kube client config rate limiting may still contribute to a noticeable drop in performance. Increasing the qps and burst values may help alleviate back pressure and improve FlytePropeller performance. An example of Kube-client-config is as follows:
legacy docs here
also, in the same docs, 
downstream-eval-duration | lower the number - lower latency and lower throughput (low throughput is because the same workflow will be evaluated more times) | This indicates how often are external events like pods completion etc recorded.
n
Yes, I saw this page 👍 Have to admit that I find it a bit confusing though. And I wonder whether this still holds when there are tens of thousands of pods, maybe lowering the refresh rate actually hurts in this scenario? We'll run experiments.
11 Views
Open in Slack
PreviousNext
Powered by