https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Björn

01/25/2023, 7:45 PM
Hello! 🙂 Anyone know of a way to get around https://github.com/flyteorg/flyte/issues/3065 on a v1.3.0 demo cluster?
k

Ketan (kumare3)

01/25/2023, 8:00 PM
by getting around you mean, i assume you dont want to increase global defaults 🙂
b

Björn

01/25/2023, 8:02 PM
Global default would be perfectly fine for a start 🙂 Can't find the config maps to change in the demo cluster
k

Ketan (kumare3)

01/26/2023, 2:13 AM
b

Björn

01/26/2023, 6:47 AM
Oh; great!! Thanks 🙂 The demo cluster has a config map
sandbox-flyte-binary-config
... But there is a bit of a mismatch between the keys... in the CM there is a key
plugins.k8s-array
- would that match the
plugins.k8s
in the documentation? I'll give it a try 🙂
Hmm... Doesn't seem to work for me, unfortunately...
k

Ketan (kumare3)

01/26/2023, 3:15 PM
Ohh you have to restart FlyteAdmin
@jeev can you help here
j

jeev

01/26/2023, 3:27 PM
I believe we specifically dropped global defaults since everyone’s environment is different.
I have a PR open for adding extra config to the sandbox here: https://github.com/flyteorg/flyte/pull/3248 Will need a flytectl change (unless you’re ok with modifying configmaps manually) as well but it will allow users to add things like resource defaults and what not, trivially. For now, modifying the existing configmap is fine, and you’ll need to “flytectl demo reload” to roll the flyte pod
@Björn to clarify, you want to increase/modify the default task resources? i can take a crack at reproducing and offering a workaround.
b

Björn

01/26/2023, 4:00 PM
Any way of increasing a task max mem limit beyond 1Gi (and hopefully lots beyond) in sandbox cluster would make it easier to sell Flyte as a solution within my org :)
j

jeev

01/26/2023, 4:05 PM
yea
one sec
Copy code
apiVersion: v1
data:
  100-overrides.yaml: |
    task_resources:
      defaults:
        cpu: 1
        memory: 2Gi
        ephemeral-storage: 10Gi
      limits:
        cpu: 16
        memory: 64Gi
        ephemeral-storage: 500Gi
kind: ConfigMap
metadata:
  name: sandbox-flyte-binary-config
  namespace: flyte
write this to a file like
patch.yaml
then:
Copy code
kubectl patch configmap sandbox-flyte-binary-config --patch-file patch.yaml
confirm that it made the change:
Copy code
kubectl get configmap sandbox-flyte-binary-config -o yaml
and you can restart flyte with:
Copy code
kubectl rollout restart deployment/sandbox-flyte-binary -n flyte
or
Copy code
flytectl demo reload
ofc change the values/key name as you see fit
the UX isn’t great right now, but we’ll update flytectl to be able to do this from a file that lives in ~/.flyte directory trivially.
b

Björn

01/26/2023, 4:16 PM
Thanks a lot! 😃 Will test it out ASAP!
j

jeev

01/26/2023, 5:46 PM
@Björn let us know if it works out 🙂
b

Björn

01/26/2023, 5:49 PM
Yay! It works! Thanks again! ❤️
b

Brian Tang

02/01/2023, 9:20 AM
@jeev what about for the sandbox cluster? i don’t see a
sandbox-flyte-binary
configmap or deployment in my sandbox environment. what should i be looking for here? this is based on
flytectl sandbox start
thanks!
Copy code
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get deployments.apps 
No resources found in default namespace.
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get deployments.apps -n flyte
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
flyte-deps-kubernetes-dashboard   1/1     1            1           56m
flyte-deps-contour-contour        1/1     1            1           56m
postgres                          1/1     1            1           56m
minio                             1/1     1            1           56m
flytepropeller                    1/1     1            1           55m
flyte-pod-webhook                 1/1     1            1           55m
flyteconsole                      1/1     1            1           55m
datacatalog                       1/1     1            1           55m
syncresources                     1/1     1            1           55m
flyteadmin                        1/1     1            1           55m
flytescheduler                    1/1     1            1           55m
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get cm 
NAME               DATA   AGE
kube-root-ca.crt   1      57m
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get cm -n flyte
NAME                               DATA   AGE
kube-root-ca.crt                   1      57m
flyte-deps-contour                 1      56m
kubernetes-dashboard-settings      0      56m
flyte-propeller-config             10     55m
flyte-admin-clusters-config        1      55m
clusterresource-template           2      55m
flyte-console-config               3      55m
flyte-scheduler-config             3      55m
flyte-clusterresourcesync-config   4      55m
datacatalog-config                 3      55m
flyte-admin-base-config            7      55m
propeller-leader                   0      55m
leader-elect                       0      56m
managed to get it working by patching
flyte-admin-base-config
instead to set a higher limit
also did a rollout restart on
flyteadmin
, not sure whether this was necessary tho
j

jeev

02/01/2023, 1:44 PM
that sounds right. the rollout restart is necessary for flyteadmin to pick up the configuration changes. there is no hot reload mechanism.
k

Ketan (kumare3)

02/01/2023, 3:11 PM
@Brian Tang can you start using the new demo cluster. It’s much lighter and faster
b

Brian Tang

02/02/2023, 9:31 AM
@Ketan (kumare3) is this
flytectl 0.6.26
? i was using 0.6.25 previously i believe. what’s the difference between
flytectl demo start
and
flytectl sandbox start
?
k

Ketan (kumare3)

02/02/2023, 3:11 PM
Demo is a single binary version of Flyte that was optimized for startup tune and better mimicking production including a container registry
We are also working on making it stateful
And you can edit single config for all Flyte
j

jeev

02/02/2023, 9:44 PM
🙌
71 Views