Team, when i am executing this command pyflyte run --remote -p flytetester --image <AWC_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest flyte/workflows/spark_example.py my_spark --triggered_date 2022-08-29 getting below exception.

Could you paste entire pod logs?

[4/4] currentAttempt done. Last Error: USER::Spark Job  Submission Failed with Error: failed to run spark-submit for SparkApplication flytetester-development/awcqz875dzn8jkhw9h5p-fyi2rkri-3: WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 092503 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/30 092503 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file 22/08/30 092503 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "awcqz875dzn8jkhw9h5p-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:589) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:526) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:492) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84) at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/08/30 092504 INFO ShutdownHookManager: Shutdown hook called 22/08/30 092504 INFO ShutdownHookManager: Deleting directory /tmp/spark-f75c423f-26c9-41d1-819e-0ee8719029ee

@Sathish kumar Venkatesan, can you set the service account to “spark” in the launch form?

@Samhita Alla i tried but same exception. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "a6nfhzsgkpddtxwrkbrg-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found.

Are you running the code on sandbox/demo?

we are running code on EKS cluster

Can you let me know the output of

kubectl get serviceaccounts -n flytetester-development

?

$ kubectl get serviceaccounts -n flytetester-development NAME SECRETS AGE default 1 70d

Oh “spark” isn’t there. Have you followed https://docs.flyte.org/en/latest/deployment/plugin_setup/k8s/index.html guide?

thanks. i will verify that link

Having a similar issue, and can confirm that

~  % kubectl -n f-sandbox get serviceaccounts
NAME      SECRETS   AGE
default   1         20h
spark     1         20h

however, its odd that the error says:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.96.0.1/api/v1/namespaces/spark-operator/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "altxd26lkmmsdv4d8hpc-n0-0-driver" is forbidden: error looking up service account spark-operator/spark: serviceaccount "spark" not found.

I'd expect it to be looking at

f-sandbox/spark

not

spark-operator/spark

@Sathish kumar Venkatesan Happy to jump on a call I think that might speed up debugging. Would that work for you?

@Yuvraj @Katrina P we were able to resolve the issue after adding service account as per the documentation link shared by @Samhita Alla thanks