https://flyte.org logo
Powered by
Title
a

Alexey Kharlamov

02/17/2023, 3:34 PM
Hi there! Recently we have updated our Flyte configuration to flyte-binary setup and faced issue with usage of default-pod-template-name configuration. flyte-template(located in flyte namespace) is listed in ConfigMap of flyte-binary(), but these template is not applied on running jobs. How to properly configure default-template in recent release?
d

David Espejo (he/him)

02/17/2023, 5:05 PM
Hi @Alexey Kharlamov If you're using the most recent Helm chart for flyte-binary, it doesn't include resource limits/requests, but you could add them to the 
inline
section like this (example): 
inline: 
    task_resources:
      defaults:
        cpu: 100m
        memory: 100Mi
        storage: 100Mi
      limits:
        memory: 1Gi
Take into account that Limits are only used by Flyte for validation so your deployed Pods could have a different behaviour. Take a look at this section of the docs https://docs.flyte.org/en/latest/concepts/admin.html#task-resource-defaults Does this answer your question?
a

Alexey Kharlamov

02/17/2023, 5:12 PM
Hi @David Espejo (he/him)! I am unable to set default-pod-template-name inside inline section of k8s plugin as stated here: https://docs.flyte.org/en/latest/deployment/configuration/general.html#set-the-default-pod-template-name-in-flytepropeller Is it possible in recent release?
inline:
    plugins:
      k8s:
        default-pod-template-name: flyte-template
Current setup inside helm chart. flyte-template defined as usual: 
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
<some settings>
d

David Espejo (he/him)

02/17/2023, 5:46 PM
I'll try to reproduce, I don't see why it wouldn't be possible Meanwhile @jeev / @Yee is 
default-pod-template-name
still configurable for propeller on single binary?
j

jeev

02/17/2023, 5:53 PM
i dont see why this wouldn't be configurable. is it possible the deployment wasn't rolled @Alexey Kharlamov?
a

Alexey Kharlamov

02/17/2023, 6:00 PM
Hi @jeev! In first message i have copied ConfigMap from deployed binary, template is listed here
j

jeev

02/17/2023, 6:26 PM
https://github.com/flyteorg/flyteplugins/blob/5981c35dbe2033e9905283c4ea4b667e79f0bf8b/go/tasks/pluginmachinery/flytek8s/config/config.go#L157
there is also a default 30s resync time
we'll have to make a minimal reproducible example in the sandbox to test
@Alexey Kharlamov: can't reproduce in sandbox: 
> cat test-pod-template.yaml
apiVersion: v1
kind: PodTemplate
metadata:
  name: test-template
  namespace: flyte
template:
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        env:
          - name: FOO
            value: BAR
> kubectl get po fb5116731e64348e4919-n0-0 -n flytesnacks-development -o jsonpath='{.spec.containers[0].env[0]}'
{"name":"FOO","value":"BAR"}%
a

Alexey Kharlamov

02/17/2023, 7:24 PM
@jeev Great! How did you add DefaultPodTemplateName? Is it possible to do from helm chart config like that? 
inline:
    plugins:
      k8s:
        default-pod-template-name: flyte-template
j

jeev

02/17/2023, 7:34 PM
inline:
      plugins:
        k8s:
          default-pod-template-name: test-template
y

Yee

02/17/2023, 7:40 PM
which version are you running do you know? could you check the version of the flyte binary image for us please?
a

Alexey Kharlamov

02/17/2023, 7:42 PM
Name:         flyte-backend-flyte-binary-config
Namespace:    flyte
Labels:       <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
              <http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
              <http://helm.sh/chart=flyte-binary-v1.3.0|helm.sh/chart=flyte-binary-v1.3.0>
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
Version 1.3
y

Yee

02/17/2023, 8:10 PM
what is 
<some settings>
?
like can you describe it a bit more? maybe copy/paste just the keys and redact the values?
a

Alexey Kharlamov

02/17/2023, 8:11 PM
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 64000Mi
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
This is full template config
I have tried to redeploy Flyte from scratch, but issue is the same=( Maybe I do smth wrong, but in previos version in worked. As seen in screen, template is not applied, memory is not mounted
d

Dan Rammer (hamersaw)

02/17/2023, 8:32 PM
@Alexey Kharlamov can you try to create a 
PodTemplate
in a more specific namespace? For example, if you are running the task in 
flytesnacks-development
then create the 
PodTemplate
there. So propeller looks for the default 
PodTemplate
in the namespace that it is running in, which defaults to 
flyte
, but perhaps in your setup this is different, or may end up being set differently. If the 
PodTemplate
works in a more specific namespace then we will know a little more.
Also, propeller should have debug level logging for 
PodTemplate
registration. Something along the lines of "registered PodTemplate 'flyte-template:flytesnacks-development' in store". Then we can make sure propeller is detecting the `PodTemplate`s
a

Alexey Kharlamov

02/17/2023, 8:36 PM
hmm, we found an issue 
E0217 20:35:26.548098       7 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1.PodTemplate: failed to list *v1.PodTemplate: podtemplates is forbidden: User "system:serviceaccount:flyte:flyte-backend-flyte-binary" cannot list resource "podtemplates" in API group "" at the cluster scope
d

Dan Rammer (hamersaw)

02/17/2023, 8:43 PM
oh yeah, that would do it 😅
a

Alexey Kharlamov

02/17/2023, 8:51 PM
After adding of ClusterRole and ClusterRoleBinding it started working. Thank you guys, you are heroes!
j

jeev

02/18/2023, 10:23 AM
ah we need to fix this in the chart - specifically add the appropriate rule to the default cluster role. the sandbox KSA is a bit more privileged to simplify adding additional service account cluster resource templates for plugins (e.g. spark). that would explain why we couldn’t reproduce on the sandbox.
y

Yee

02/23/2023, 5:40 AM
@jeev
6 Views
#ask-the-communityJoin Slack
Powered by