Thread
#announcements
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    Team, when i am executing this command pyflyte run --remote -p flytetester --image <AWC_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com/flyte-pyspark:latest flyte/workflows/spark_example.py my_spark --triggered_date 2022-08-29 getting below exception.
    Kevin Su

    Kevin Su

    4 weeks ago
    Could you paste entire pod logs?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    [4/4] currentAttempt done. Last Error: USER::Spark Job  Submission Failed with Error: failed to run spark-submit for SparkApplication flytetester-development/awcqz875dzn8jkhw9h5p-fyi2rkri-3: WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file😕opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 22/08/30 09:25:02 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 09:25:02 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 09:25:02 WARN SparkConf: The configuration key 'spark.blacklist.timeout' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.timeout 22/08/30 09:25:02 WARN SparkConf: The configuration key 'spark.blacklist.enabled' has been deprecated as of Spark 3.1.0 and may be removed in the future. Please use spark.excludeOnFailure.enabled 22/08/30 09:25:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/08/30 09:25:03 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file 22/08/30 09:25:03 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "awcqz875dzn8jkhw9h5p-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found. at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:589) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:526) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:492) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84) at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207) at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207) at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/08/30 09:25:04 INFO ShutdownHookManager: Shutdown hook called 22/08/30 09:25:04 INFO ShutdownHookManager: Deleting directory /tmp/spark-f75c423f-26c9-41d1-819e-0ee8719029ee
    Samhita Alla

    Samhita Alla

    4 weeks ago
    @Sathish kumar Venkatesan, can you set the service account to “spark” in the launch form?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    @Samhita Alla i tried but same exception. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.20.0.1/api/v1/namespaces/flytetester-development/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "a6nfhzsgkpddtxwrkbrg-fyi2rkri-3-driver" is forbidden: error looking up service account flytetester-development/spark: serviceaccount "spark" not found.
    Samhita Alla

    Samhita Alla

    4 weeks ago
    Are you running the code on sandbox/demo?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    we are running code on EKS cluster
    Samhita Alla

    Samhita Alla

    4 weeks ago
    Can you let me know the output of
    kubectl get serviceaccounts -n flytetester-development
    ?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    $ kubectl get serviceaccounts -n flytetester-development NAME SECRETS AGE default 1 70d
    Samhita Alla

    Samhita Alla

    4 weeks ago
    Oh “spark” isn’t there. Have you followed https://docs.flyte.org/en/latest/deployment/plugin_setup/k8s/index.html guide?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    4 weeks ago
    thanks. i will verify that link
    k

    Katrina P

    3 weeks ago
    Having a similar issue, and can confirm that
    ~  % kubectl -n f-sandbox get serviceaccounts
    NAME      SECRETS   AGE
    default   1         20h
    spark     1         20h
    however, its odd that the error says:
    Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.96.0.1/api/v1/namespaces/spark-operator/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "altxd26lkmmsdv4d8hpc-n0-0-driver" is forbidden: error looking up service account spark-operator/spark: serviceaccount "spark" not found.
    I'd expect it to be looking at
    f-sandbox/spark
    not
    spark-operator/spark
    if I set the service account to one specifically in the
    spark-operator
    namespace I get
    [1/1] currentAttempt done. Last Error: USER::Spark Job Failed with Error: driver pod not found
    Will continue to debug this tomorrow, but just in case there's a solution here to Sathish's issue I'd be interested in case it can help me debug my issue (even if the root cause is different)
    y

    Yuvraj

    3 weeks ago
    @Sathish kumar Venkatesan Happy to jump on a call I think that might speed up debugging. Would that work for you?
    Sathish kumar Venkatesan

    Sathish kumar Venkatesan

    3 weeks ago
    @Yuvraj @Katrina P we were able to resolve the issue after adding service account as per the documentation link shared by @Samhita Alla thanks