Hi, I'm new to flyte and kubernetes. I followed th...
# flyte-deployment
r
Hi, I'm new to flyte and kubernetes. I followed the flyte the hard way tutorial, and was able to get the deployment up, with an ingress through an ALB. However, when I try to run the hello_world example, a pod is created, but then fails to proceed further. Here's the event output for `kubectl describe pod <podname> -n flytesnacks-development`:
Copy code
Events:
Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  67s   default-scheduler  0/4 nodes are available: 4 Insufficient cpu. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.
What would be your recommendation course of action for de-bugging this problem?
d
what instance type are you using? also, you can reduce the default CPU requests from the Helm values file:
Copy code
task_resources:
      defaults:
        cpu: 500m
I'm glad the tutorial got you there, and I'm really interested in learning from this to adjust the values to work OOB
r
Thanks David. Instance type that is currently being created are t3.medium. I'll try reducing the default CPU requests. It looks like I'm not specifying
task_resources
, so will add that to the Helm values file.
I've dropped the default to 100m, ran
helm upgrade...
, and then re-ran the tasks. No change in behavior (same error FailedScheduling observed).
k
cc @David Espejo (he/him) can you help here? It seems like the nodes are not autoscaling or the node configurations is wrong, that we do not have nodes that can have 100Mi cpu?
@Ryo Kita what
pod-name
is this?
r
The pod name is
a24gnswfz78p92ggx8jh-fvub2h2y-0
is that what you're asking?
d
@Ketan (kumare3) I'm talking to Ryo, while there's no documented resource consumption profiling for Flyte AFAIK, I've seen
t3.medium
(2vCPU) not being enough in the past. We'll test with
t3.xlarge
r
Following up here. The switch to
t3.xlarge
worked. Thank you @David Espejo (he/him) for your help.