https://flyte.org logo
#announcements
Title
# announcements
s

Sathish kumar Venkatesan

08/30/2022, 6:39 AM
Team, when i am executing this command pyflyte run --remote -p flytetester --image <AWC_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest flyte/workflows/spark_example.py my_spark --triggered_date 2022-08-29 getting below exception.
k

Kevin Su

08/30/2022, 8:42 AM
Could you paste entire pod logs?
s

Sathish kumar Venkatesan

08/30/2022, 9:27 AM
[4/4] currentAttempt done. Last Error: USER::Spark Job  Submission Failed with Error: failed to run spark-submit for SparkApplication flytetester-development/awcqz875dzn8jkhw9h5p-fyi2rkri-3: WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 092503 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/30 092503 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file 22/08/30 092503 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "awcqz875dzn8jkhw9h5p-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:589) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:526) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:492) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84) at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/08/30 092504 INFO ShutdownHookManager: Shutdown hook called 22/08/30 092504 INFO ShutdownHookManager: Deleting directory /tmp/spark-f75c423f-26c9-41d1-819e-0ee8719029ee
s

Samhita Alla

08/30/2022, 9:39 AM
@Sathish kumar Venkatesan, can you set the service account to “spark” in the launch form?
s

Sathish kumar Venkatesan

08/30/2022, 9:50 AM
@Samhita Alla i tried but same exception. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "a6nfhzsgkpddtxwrkbrg-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found.
s

Samhita Alla

08/30/2022, 9:51 AM
Are you running the code on sandbox/demo?
s

Sathish kumar Venkatesan

08/30/2022, 9:54 AM
we are running code on EKS cluster
s

Samhita Alla

08/30/2022, 10:03 AM
Can you let me know the output of
kubectl get serviceaccounts -n flytetester-development
?
s

Sathish kumar Venkatesan

08/30/2022, 10:08 AM
$ kubectl get serviceaccounts -n flytetester-development NAME SECRETS AGE default 1 70d
s

Samhita Alla

08/30/2022, 10:08 AM
Oh “spark” isn’t there. Have you followed https://docs.flyte.org/en/latest/deployment/plugin_setup/k8s/index.html guide?
s

Sathish kumar Venkatesan

08/30/2022, 10:35 AM
thanks. i will verify that link
👍 1
k

Katrina P

08/30/2022, 11:18 PM
Having a similar issue, and can confirm that
Copy code
~  % kubectl -n f-sandbox get serviceaccounts
NAME      SECRETS   AGE
default   1         20h
spark     1         20h
however, its odd that the error says:
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.96.0.1/api/v1/namespaces/spark-operator/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "altxd26lkmmsdv4d8hpc-n0-0-driver" is forbidden: error looking up service account spark-operator/spark: serviceaccount "spark" not found.
I'd expect it to be looking at
f-sandbox/spark
not
spark-operator/spark
if I set the service account to one specifically in the
spark-operator
namespace I get
[1/1] currentAttempt done. Last Error: USER::Spark Job Failed with Error: driver pod not found
Will continue to debug this tomorrow, but just in case there's a solution here to Sathish's issue I'd be interested in case it can help me debug my issue (even if the root cause is different)
y

Yuvraj

09/01/2022, 5:21 AM
@Sathish kumar Venkatesan Happy to jump on a call I think that might speed up debugging. Would that work for you?
s

Sathish kumar Venkatesan

09/02/2022, 7:49 AM
@Yuvraj @Katrina P we were able to resolve the issue after adding service account as per the documentation link shared by @Samhita Alla thanks
75 Views