https://flyte.org
Join Slack
Hello! :slightly_smiling_face: Anyone know of a way to get around <https://github.com/flyteorg/flyte...
# flyte-support
q
Hello! 🙂 Anyone know of a way to get around https://github.com/flyteorg/flyte/issues/3065 on a v1.3.0 demo cluster?
f
by getting around you mean, i assume you dont want to increase global defaults 🙂
q
Global default would be perfectly fine for a start 🙂 Can't find the config maps to change in the demo cluster
f
q
Oh; great!! Thanks 🙂 The demo cluster has a config map 
sandbox-flyte-binary-config
... But there is a bit of a mismatch between the keys... in the CM there is a key 
plugins.k8s-array
- would that match the 
plugins.k8s
in the documentation? I'll give it a try 🙂
Hmm... Doesn't seem to work for me, unfortunately...
f
Ohh you have to restart FlyteAdmin
@freezing-boots-56761 can you help here
f
I believe we specifically dropped global defaults since everyone’s environment is different.
I have a PR open for adding extra config to the sandbox here: https://github.com/flyteorg/flyte/pull/3248 Will need a flytectl change (unless you’re ok with modifying configmaps manually) as well but it will allow users to add things like resource defaults and what not, trivially. For now, modifying the existing configmap is fine, and you’ll need to “flytectl demo reload” to roll the flyte pod
@quick-salesclerk-18019 to clarify, you want to increase/modify the default task resources? i can take a crack at reproducing and offering a workaround.
❤️ 1
🔥 1
q
Any way of increasing a task max mem limit beyond 1Gi (and hopefully lots beyond) in sandbox cluster would make it easier to sell Flyte as a solution within my org :)
f
yea
one sec
Copy code
apiVersion: v1
data:
  100-overrides.yaml: |
    task_resources:
      defaults:
        cpu: 1
        memory: 2Gi
        ephemeral-storage: 10Gi
      limits:
        cpu: 16
        memory: 64Gi
        ephemeral-storage: 500Gi
kind: ConfigMap
metadata:
  name: sandbox-flyte-binary-config
  namespace: flyte
write this to a file like 
patch.yaml
then:
Copy code
kubectl patch configmap sandbox-flyte-binary-config --patch-file patch.yaml
confirm that it made the change:
Copy code
kubectl get configmap sandbox-flyte-binary-config -o yaml
and you can restart flyte with:
Copy code
kubectl rollout restart deployment/sandbox-flyte-binary -n flyte
or
Copy code
flytectl demo reload
ofc change the values/key name as you see fit
the UX isn’t great right now, but we’ll update flytectl to be able to do this from a file that lives in ~/.flyte directory trivially.
q
Thanks a lot! 😃 Will test it out ASAP!
👍 1
f
@quick-salesclerk-18019 let us know if it works out 🙂
q
Yay! It works! Thanks again! ❤️
🙌 1
m
@freezing-boots-56761 what about for the sandbox cluster? i don’t see a 
sandbox-flyte-binary
configmap or deployment in my sandbox environment. what should i be looking for here? this is based on 
flytectl sandbox start
thanks!
Copy code
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get deployments.apps 
No resources found in default namespace.
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get deployments.apps -n flyte
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
flyte-deps-kubernetes-dashboard   1/1     1            1           56m
flyte-deps-contour-contour        1/1     1            1           56m
postgres                          1/1     1            1           56m
minio                             1/1     1            1           56m
flytepropeller                    1/1     1            1           55m
flyte-pod-webhook                 1/1     1            1           55m
flyteconsole                      1/1     1            1           55m
datacatalog                       1/1     1            1           55m
syncresources                     1/1     1            1           55m
flyteadmin                        1/1     1            1           55m
flytescheduler                    1/1     1            1           55m
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get cm 
NAME               DATA   AGE
kube-root-ca.crt   1      57m
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get cm -n flyte
NAME                               DATA   AGE
kube-root-ca.crt                   1      57m
flyte-deps-contour                 1      56m
kubernetes-dashboard-settings      0      56m
flyte-propeller-config             10     55m
flyte-admin-clusters-config        1      55m
clusterresource-template           2      55m
flyte-console-config               3      55m
flyte-scheduler-config             3      55m
flyte-clusterresourcesync-config   4      55m
datacatalog-config                 3      55m
flyte-admin-base-config            7      55m
propeller-leader                   0      55m
leader-elect                       0      56m
managed to get it working by patching 
flyte-admin-base-config
instead to set a higher limit
🙌 1
also did a rollout restart on 
flyteadmin
, not sure whether this was necessary tho
f
that sounds right. the rollout restart is necessary for flyteadmin to pick up the configuration changes. there is no hot reload mechanism.
👍 1
f
@millions-table-8574 can you start using the new demo cluster. It’s much lighter and faster
m
@freezing-airport-6809 is this 
flytectl 0.6.26
? i was using 0.6.25 previously i believe. what’s the difference between 
flytectl demo start
and 
flytectl sandbox start
?
f
Demo is a single binary version of Flyte that was optimized for startup tune and better mimicking production including a container registry
We are also working on making it stateful
And you can edit single config for all Flyte
👀 1
f
🙌
155 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by