I ve deployed flyte using kubeadm along with spark k8s opera

Question

I ve deployed flyte using kubeadm along with spark k8s operator which run successful did try run the example from and got some issues logs in thread Someone know about `Non spark on k8s command provided`

Rodrigo Baron · Answer

```++ id u + myuid=0 ++ id g + mygid=0 + set +e ++ getent passwd 0 + uidentry=root x 0 0 root root bin bash + set e + z root x 0 0 root root bin bash + SPARK CLASSPATH= opt spark jars + env + grep SPARK JAVA OPT + sort t k4 n + sed s ^= = 1 g + readarray t SPARK EXECUTOR JAVA OPTS + n + == 2 + == 3 + n + z + case $1 in + echo Non spark on k8s command provided proceeding in pass through mode + CMD= $ Non spark on k8s command provided proceeding in pass through mode + exec usr bin tini s pyflyte execute inputs output prefix raw output data prefix resolver flytekit core python auto container default task resolver task module k8s spark pyspark pi task name hello spark Welcome to Flyte Version 0 22 2 Attempting to run with flytekit core python auto container default task resolver WARNING root No config file provided or invalid flyte config file path flytekit config specified Using user directory tmp flyte 20220204 165445 sandbox local flytekit a504ea1e3e3c62771228ed2d83186827 asctime 2022 02 04 16 54 57 721 name flytekit levelname DEBUG message Task returns unnamed native tuple lt class float gt DEBUG flytekit Task returns unnamed native tuple lt class float gt asctime 2022 02 04 16 54 57 722 name flytekit levelname DEBUG message Task returns unnamed native tuple lt class int gt DEBUG flytekit Task returns unnamed native tuple lt class int gt asctime 2022 02 04 16 54 57 821 name flytekit levelname DEBUG message Task returns unnamed native tuple lt class float gt DEBUG flytekit Task returns unnamed native tuple lt class float gt No images specified will use the default image Running native typed task INFO root Entering timed context Copying gt tmp flyte7c4gkp2a local flytekit inputs pb INFO root Output of command aws endpoint url s3 cp tmp flyte7c4gkp2a local flytekit inputs pb b Completed 22 Bytes 22 Bytes 248 Bytes s with 1 file s remaining rdownload to tmp flyte7c4gkp2a local flytekit inputs pb n INFO root Exiting timed context Copying gt tmp flyte7c4gkp2a local flytekit inputs pb Wall Time 9 195532751000428s Process Time 0 0055688540000000675s 22 02 04 16 56 07 WARN NativeCodeLoader Unable to load native hadoop library for your platform using builtin java classes where applicable Using Spark s default log4j profile org apache spark log4j defaults properties Setting default log level to WARN To adjust logging level use sc setLogLevel newLevel For SparkR use setLogLevel newLevel INFO py4j java gateway Error while receiving Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1207 in send command raise Py4JNetworkError Answer from Java side is empty py4j protocol Py4JNetworkError Answer from Java side is empty ERROR root Exception while sending command Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1207 in send command raise Py4JNetworkError Answer from Java side is empty py4j protocol Py4JNetworkError Answer from Java side is empty During handling of the above exception another exception occurred Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1033 in send command response = connection send command command File opt venv lib python3 8 site packages py4j java gateway py line 1211 in send command raise Py4JNetworkError py4j protocol Py4JNetworkError Error while receiving ERROR root Begin System Error Captured by Flyte ERROR root Traceback most recent call last File opt venv lib python3 8 site packages flytekit common exceptions scopes py line 165 in system entry point return wrapped args kwargs File opt venv lib python3 8 site packages flytekit core base task py line 442 in dispatch execute new user params = self pre execute ctx user space params File opt venv lib python3 8 site packages flytekitplugins spark task py line 122 in pre execute self sess = sess builder getOrCreate File opt venv lib python3 8 site packages pyspark sql session py line 228 in getOrCreate sc = SparkContext getOrCreate sparkConf File opt venv lib python3 8 site packages pyspark context py line 384 in getOrCreate SparkContext conf=conf or SparkConf File opt venv lib python3 8 site packages pyspark context py line 146 in init self do init master appName sparkHome pyFiles environment batchSize serializer File opt venv lib python3 8 site packages pyspark context py line 209 in do init self jsc = jsc or self initialize context self conf jconf File opt venv lib python3 8 site packages pyspark context py line 321 in initialize context return self jvm JavaSparkContext jconf File opt venv lib python3 8 site packages py4j java gateway py line 1568 in call return value = get return value File opt venv lib python3 8 site packages py4j protocol py line 334 in get return value raise Py4JError Message An error occurred while calling None org apache spark api java JavaSparkContext SYSTEM ERROR Contact platform administrators ERROR root End Error Captured by Flyte INFO root Entering timed context Writing tmp flyte7c4gkp2a local flytekit engine dir gt INFO root Output of command aws endpoint url s3 cp recursive acl bucket owner full control tmp flyte7c4gkp2a local flytekit engine dir b Completed 1 7 KiB 1 7 KiB 291 7 KiB s with 1 file s remaining rupload tmp flyte7c4gkp2a local flytekit engine dir error pb to n INFO root Exiting timed context Writing tmp flyte7c4gkp2a local flytekit engine dir gt Wall Time 11 491473693000444s Process Time 0 009896259000000018s INFO root Engine folder written successfully to the output prefix ERROR py4j java gateway An error occurred while trying to connect to the Java server 127 0 0 1 37755 Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 977 in get connection connection = self deque pop IndexError pop from an empty deque During handling of the above exception another exception occurred Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1115 in start self socket connect self address self port ConnectionRefusedError Errno 111 Connection refused ERROR py4j java gateway An error occurred while trying to connect to the Java server 127 0 0 1 37755 Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 977 in get connection connection = self deque pop IndexError pop from an empty deque During handling of the above exception another exception occurred Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1115 in start self socket connect self address self port ConnectionRefusedError Errno 111 Connection refused ERROR py4j java gateway An error occurred while trying to connect to the Java server 127 0 0 1 37755 Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 977 in get connection connection = self deque pop IndexError pop from an empty deque During handling of the above exception another exception occurred Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1115 in start self socket connect self address self port ConnectionRefusedError Errno 111 Connection refused ERROR py4j java gateway An error occurred while trying to connect to the Java server 127 0 0 1 37755 Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 977 in get connection connection = self deque pop IndexError pop from an empty deque During handling of the above exception another exception occurred Traceback most recent call last File opt venv lib python3 8 site packages py4j java gateway py line 1115 in start self socket connect self address self port ConnectionRefusedError Errno 111 Connection refused```

Haytham Abuelfutuh · Answer

< Yee> < Eduardo Apolinario eapolinario > would you be able to help < Rodrigo Baron>

Yee · Answer

hey < Rodrigo Baron> when you get a chance could you send us the pod specs for the spark operator you have running and the task pod that failed

Yee · Answer

i ll try that on my end as well and see what the differences are

Yee · Answer

from there hopefully we can try to narrow it down

Yee · Answer

i m not familiar with the setup prescribed by the py pi yaml files not that i m terribly familiar with the helm chart we use either but at least that s one difference we can start to dig into

Rodrigo Baron · Answer

the pod is running the flyte task ```apiVersion v1 kind Pod metadata annotations false 501668ab8335b3661174264ea733d6a0589a733172a8ae3a388d8e99d434abfa 172 29 77 31 32 172 29 77 31 32 creationTimestamp 2022 02 04T21 01 39Z labels domain development execution id byn0i5mak6 interruptible false node id k8ssparkpysparkpihellospark project flytesnacks shard key 11 task name k8s spark pyspark pi hello spark workflow name flytegen k8s spark pyspark pi hello spark name byn0i5mak6 f3cpp6wq 0 namespace flytesnacks development ownerReferences apiVersion blockOwnerDeletion true controller true kind flyteworkflow name byn0i5mak6 uid a0c7d530 8416 40de 9a70 11024bc7a7a6 resourceVersion 1081142 uid 731da94e 2786 4d23 9366 ebe7a841f62e spec containers args pyflyte execute inputs output prefix raw output data prefix resolver flytekit core python auto container default task resolver task module k8s spark pyspark pi task name hello spark env name FLYTE INTERNAL IMAGE value rodrigobaron flyte 0 0 4 name FLYTE INTERNAL EXECUTION WORKFLOW value flytesnacks development flytegen k8s spark pyspark pi hello spark name FLYTE INTERNAL EXECUTION ID value byn0i5mak6 name FLYTE INTERNAL EXECUTION PROJECT value flytesnacks name FLYTE INTERNAL EXECUTION DOMAIN value development name FLYTE ATTEMPT NUMBER value 0 name FLYTE INTERNAL TASK PROJECT value flytesnacks name FLYTE INTERNAL TASK DOMAIN value development name FLYTE INTERNAL TASK NAME value k8s spark pyspark pi hello spark name FLYTE INTERNAL TASK VERSION value v1 name FLYTE INTERNAL PROJECT value flytesnacks name FLYTE INTERNAL DOMAIN value development name FLYTE INTERNAL NAME value k8s spark pyspark pi hello spark name FLYTE INTERNAL VERSION value v1 name FLYTE AWS ENDPOINT value name FLYTE AWS ACCESS KEY ID value minio name FLYTE AWS SECRET ACCESS KEY value miniostorage image rodrigobaron flyte 0 0 4 imagePullPolicy IfNotPresent name byn0i5mak6 f3cpp6wq 0 resources limits cpu 100m memory 200Mi requests cpu 100m memory 200Mi terminationMessagePath dev termination log terminationMessagePolicy FallbackToLogsOnError volumeMounts mountPath var run secrets kubernetes io serviceaccount name kube api access hg2dw readOnly true dnsPolicy ClusterFirst enableServiceLinks true nodeName k8s preemptionPolicy PreemptLowerPriority priority 0 restartPolicy Never schedulerName default scheduler securityContext serviceAccount default serviceAccountName default terminationGracePeriodSeconds 30 tolerations effect NoExecute key operator Exists tolerationSeconds 300 effect NoExecute key operator Exists tolerationSeconds 300 volumes name kube api access hg2dw projected defaultMode 420 sources serviceAccountToken expirationSeconds 3607 path token configMap items key ca crt path ca crt name kube root ca crt downwardAPI items fieldRef apiVersion v1 fieldPath metadata namespace path namespace status conditions lastProbeTime null lastTransitionTime 2022 02 04T21 01 39Z status True type Initialized lastProbeTime null lastTransitionTime 2022 02 04T21 01 42Z status True type Ready lastProbeTime null lastTransitionTime 2022 02 04T21 01 42Z status True type ContainersReady lastProbeTime null lastTransitionTime 2022 02 04T21 01 39Z status True type PodScheduled containerStatuses containerID 4 image rodrigobaron flyte 0 0 4 imageID lastState name byn0i5mak6 f3cpp6wq 0 ready true restartCount 0 started true state running startedAt 2022 02 04T21 01 41Z hostIP 192 168 0 222 phase Running podIP 172 29 77 31 podIPs ip 172 29 77 31 qosClass Guaranteed startTime 2022 02 04T21 01 39Z ```

Rodrigo Baron · Answer

the pod is running the spark operator ```apiVersion v1 kind Pod metadata annotations f4e801034ed42fe8954ef4a40ae13261719fc332ef14cd349e6ab039433fa365 172 29 77 32 32 172 29 77 32 32 metrics 10254 true creationTimestamp 2022 02 04T14 17 10Z generateName flyte sparkoperator 86bc9b4dc9 labels flyte sparkoperator pod template hash 86bc9b4dc9 name flyte sparkoperator 86bc9b4dc9 npgxp namespace default ownerReferences apiVersion apps v1 blockOwnerDeletion true controller true kind ReplicaSet name flyte sparkoperator 86bc9b4dc9 uid 71878b96 d5d4 473a 91bf 8b31488b3427 resourceVersion 997418 uid 10d48af7 cb93 43d9 8be0 850e27ef167a spec containers args v=2 logtostderr namespace= ingress url format= controller threads=10 resync interval=30 enable batch scheduler=false enable metrics=true metrics labels=app type metrics port=10254 metrics endpoint= metrics metrics prefix= enable resource quota enforcement=false image imagePullPolicy IfNotPresent name sparkoperator ports containerPort 10254 name metrics protocol TCP resources limits cpu 100m memory 300Mi requests cpu 100m memory 300Mi securityContext terminationMessagePath dev termination log terminationMessagePolicy File volumeMounts mountPath var run secrets kubernetes io serviceaccount name kube api access ppdjt readOnly true dnsPolicy ClusterFirst enableServiceLinks true nodeName k8s preemptionPolicy PreemptLowerPriority priority 0 restartPolicy Always schedulerName default scheduler securityContext serviceAccount flyte sparkoperator serviceAccountName flyte sparkoperator terminationGracePeriodSeconds 30 tolerations effect NoExecute key operator Exists tolerationSeconds 300 effect NoExecute key operator Exists tolerationSeconds 300 volumes name kube api access ppdjt projected defaultMode 420 sources serviceAccountToken expirationSeconds 3607 path token configMap items key ca crt path ca crt name kube root ca crt downwardAPI items fieldRef apiVersion v1 fieldPath metadata namespace path namespace status conditions lastProbeTime null lastTransitionTime 2022 02 04T14 17 10Z status True type Initialized lastProbeTime null lastTransitionTime 2022 02 04T14 17 14Z status True type Ready lastProbeTime null lastTransitionTime 2022 02 04T14 17 14Z status True type ContainersReady lastProbeTime null lastTransitionTime 2022 02 04T14 17 10Z status True type PodScheduled containerStatuses containerID 4 image imageID lastState name sparkoperator ready true restartCount 0 started true state running startedAt 2022 02 04T14 17 14Z hostIP 192 168 0 222 phase Running podIP 172 29 77 32 podIPs ip 172 29 77 32 qosClass Guaranteed startTime 2022 02 04T14 17 10Z ```

Rodrigo Baron · Answer

I put all in a repository <https github com rodrigobaron flyte spark op>

Yee · Answer

and remind me what version of flytekit you have

Yee · Answer

oh and also any obvious logs in the spark operator pod

Yee · Answer

like anything erroring etc

Yee · Answer

just looking at the operator pod not really seeing any differences the resources you have seem a bit low but that s it

Rodrigo Baron · Answer

if increase the memory the task run successfully however don t have any new logs in the spark operator pod about the spark job

Rodrigo Baron · Answer

the most obvious thing is this log ```+ readarray t SPARK EXECUTOR JAVA OPTS + n + == 2 + == 3 + n + z + case $1 in + echo Non spark on k8s command provided proceeding in pass through mode ```

Rodrigo Baron · Answer

i m using the lastest version of flytekit 0 26 1

Rodrigo Baron · Answer

hoo is `spark` missing in my conf ```enabled plugins ```

Rodrigo Baron · Answer

this is the <https github com rodrigobaron flyte spark op commit ab5bc5d9c0940bffca279fa227e10aadaa55b7d8|fix> thanks for your support slightly smiling face

Yee · Answer

sorry about this glad you figured it out

Yee · Answer

didn t even consider this definitely should have

Yee · Answer

yeah it was weird that you were able to get the pod spec typically what happens is that the worker pods are killed immediately by the driver