https://flyte.org
Join Slack
Hi there! Recently we have updated our Flyte configuration to flyte-binary setup and faced issue wi...
# flyte-support
w
Hi there! Recently we have updated our Flyte configuration to flyte-binary setup and faced issue with usage of default-pod-template-name configuration. flyte-template(located in flyte namespace) is listed in ConfigMap of flyte-binary(), but these template is not applied on running jobs. How to properly configure default-template in recent release?
a
Hi @white-yak-77603 If you're using the most recent Helm chart for flyte-binary, it doesn't include resource limits/requests, but you could add them to the 
inline
section like this (example):
Copy code
inline: 
    task_resources:
      defaults:
        cpu: 100m
        memory: 100Mi
        storage: 100Mi
      limits:
        memory: 1Gi
Take into account that Limits are only used by Flyte for validation so your deployed Pods could have a different behaviour. Take a look at this section of the docs https://docs.flyte.org/en/latest/concepts/admin.html#task-resource-defaults Does this answer your question?
w
Hi @average-finland-92144! I am unable to set default-pod-template-name inside inline section of k8s plugin as stated here: https://docs.flyte.org/en/latest/deployment/configuration/general.html#set-the-default-pod-template-name-in-flytepropeller Is it possible in recent release?
Copy code
inline:
    plugins:
      k8s:
        default-pod-template-name: flyte-template
Current setup inside helm chart. flyte-template defined as usual:
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
<some settings>
a
I'll try to reproduce, I don't see why it wouldn't be possible Meanwhile @freezing-boots-56761 / @thankful-minister-83577 is 
default-pod-template-name
still configurable for propeller on single binary?
f
i dont see why this wouldn't be configurable. is it possible the deployment wasn't rolled @white-yak-77603?
w
Hi @freezing-boots-56761! In first message i have copied ConfigMap from deployed binary, template is listed here
f
there is also a default 30s resync time
we'll have to make a minimal reproducible example in the sandbox to test
@white-yak-77603: can't reproduce in sandbox:
Copy code
> cat test-pod-template.yaml
apiVersion: v1
kind: PodTemplate
metadata:
  name: test-template
  namespace: flyte
template:
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        env:
          - name: FOO
            value: BAR
Copy code
> kubectl get po fb5116731e64348e4919-n0-0 -n flytesnacks-development -o jsonpath='{.spec.containers[0].env[0]}'
{"name":"FOO","value":"BAR"}%
w
@freezing-boots-56761 Great! How did you add DefaultPodTemplateName? Is it possible to do from helm chart config like that?
Copy code
inline:
    plugins:
      k8s:
        default-pod-template-name: flyte-template
f
Copy code
inline:
      plugins:
        k8s:
          default-pod-template-name: test-template
t
which version are you running do you know? could you check the version of the flyte binary image for us please?
w
Copy code
Name:         flyte-backend-flyte-binary-config
Namespace:    flyte
Labels:       <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
              <http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
              <http://helm.sh/chart=flyte-binary-v1.3.0|helm.sh/chart=flyte-binary-v1.3.0>
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
Version 1.3
t
what is 
<some settings>
?
like can you describe it a bit more? maybe copy/paste just the keys and redact the values?
w
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 64000Mi
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
This is full template config
I have tried to redeploy Flyte from scratch, but issue is the same=( Maybe I do smth wrong, but in previos version in worked. As seen in screen, template is not applied, memory is not mounted
h
@white-yak-77603 can you try to create a 
PodTemplate
in a more specific namespace? For example, if you are running the task in 
flytesnacks-development
then create the 
PodTemplate
there. So propeller looks for the default 
PodTemplate
in the namespace that it is running in, which defaults to 
flyte
, but perhaps in your setup this is different, or may end up being set differently. If the 
PodTemplate
works in a more specific namespace then we will know a little more.
Also, propeller should have debug level logging for 
PodTemplate
registration. Something along the lines of "registered PodTemplate 'flyte-template:flytesnacks-development' in store". Then we can make sure propeller is detecting the `PodTemplate`s
w
hmm, we found an issue
Copy code
E0217 20:35:26.548098       7 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1.PodTemplate: failed to list *v1.PodTemplate: podtemplates is forbidden: User "system:serviceaccount:flyte:flyte-backend-flyte-binary" cannot list resource "podtemplates" in API group "" at the cluster scope
🚀 1
h
oh yeah, that would do it 😅
w
After adding of ClusterRole and ClusterRoleBinding it started working. Thank you guys, you are heroes!
🙌🏽 1
🙌 1
❤️ 1
f
ah we need to fix this in the chart - specifically add the appropriate rule to the default cluster role. the sandbox KSA is a bit more privileged to simplify adding additional service account cluster resource templates for plugins (e.g. spark). that would explain why we couldn’t reproduce on the sandbox.
✅ 1
t
@freezing-boots-56761
168 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by