Sathish kumar Venkatesan
08/30/2022, 6:39 AMKevin Su
08/30/2022, 8:42 AMSathish kumar Venkatesan
08/30/2022, 9:27 AM[4/4] currentAttempt done. Last Error: USER::Spark Job Submission Failed with Error: failed to run spark-submit for SparkApplication flytetester-development/awcqz875dzn8jkhw9h5p-fyi2rkri-3: WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout
22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled
22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout
22/08/30 092502 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled
22/08/30 092503 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/08/30 092503 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
22/08/30 092503 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "awcqz875dzn8jkhw9h5p-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:589)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:526)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:492)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84)
at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/08/30 092504 INFO ShutdownHookManager: Shutdown hook called
22/08/30 092504 INFO ShutdownHookManager: Deleting directory /tmp/spark-f75c423f-26c9-41d1-819e-0ee8719029eeSamhita Alla
Sathish kumar Venkatesan
08/30/2022, 9:50 AMSamhita Alla
Sathish kumar Venkatesan
08/30/2022, 9:54 AMSamhita Alla
kubectl get serviceaccounts -n flytetester-development
?Sathish kumar Venkatesan
08/30/2022, 10:08 AMSamhita Alla
Sathish kumar Venkatesan
08/30/2022, 10:35 AMKatrina P
08/30/2022, 11:18 PM~ % kubectl -n f-sandbox get serviceaccounts
NAME SECRETS AGE
default 1 20h
spark 1 20h
however, its odd that the error says:
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.96.0.1/api/v1/namespaces/spark-operator/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "altxd26lkmmsdv4d8hpc-n0-0-driver" is forbidden: error looking up service account spark-operator/spark: serviceaccount "spark" not found.
I'd expect it to be looking at f-sandbox/spark
not spark-operator/spark
spark-operator
namespace I get [1/1] currentAttempt done. Last Error: USER::Spark Job Failed with Error: driver pod not found
Yuvraj
09/01/2022, 5:21 AMSathish kumar Venkatesan
09/02/2022, 7:49 AM