https://flyte.org
Join Slack
Hey everyone At large scale(hundreds of paralel tasks), we are getting 503 errors. what can be done?
# flyte-support
b
Hey everyone At large scale(hundreds of paralel tasks), we are getting 503 errors. what can be done?
f
Is this single binary?
b
these are python tasks
f
I mean Flyte deployment
b
I'll ask our devops tommorow @tall-ram-83532. can you give me some context on why you are asking that?
f
yes, the reason is, single binary deployment is a little low powered
you can try increasing the memory / cpu for the deployment
this will help, but if it does not, you will have to deploy the flyte-core helm chart
b
thanks @freezing-airport-6809
t
Thanks @freezing-airport-6809, I can confirm that we are in fact using flyte-binary, will implement your suggestions 🙏
💯 1
f
Please increase resources
First
👍 1
t
@freezing-airport-6809 Is there an example of how to increase resources in flyte-binary? I'm guessing it's the cluster_resources section within configuration.inline but the documentation does not have any more specifics
NVM, found it, here's what I updated to:
Copy code
deployment:
  image:
    tag: "v1.10.6"
  resources:
    requests:
      cpu: 0.5
      memory: 20Gi
    limits:
      cpu: 2
      memory: 32Gi
f
Can you please increase the CPU - 0.5 is nothing. Atleast give it 2. It does a lot of v things right
👍 1
4 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by