Katrina P
04/21/2022, 4:33 AMflytectl register files --config ~/.flyte/config.yaml <https://github.com/flyteorg/flytesnacks/releases/download/v0.3.70/snacks-cookbook-integrations-kubernetes-k8s_spark.tar.gz> --archive -p flytesnacks -d development --version latest
(I can run the examples locally)
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/0> --raw-output-data-prefix <s3://my-s3-bucket/72/kdu1tzd712-n0-0> --checkpoint-path <s3://my-s3-bucket/72/kdu1tzd712-n0-0/_flytecheckpoints> --prev-checkpoint '""' --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.dataframe_passing task-name create_spark_df
22/04/21 04:27:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{"asctime": "2022-04-21 04:28:13,716", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n return wrapped(*args, **kwargs)\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 463, in dispatch_execute\n new_user_params = self.pre_execute(ctx.user_space_params)\n File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 124, in pre_execute\n self.sess = sess_builder.getOrCreate()\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n sc = SparkContext.getOrCreate(sparkConf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n SparkContext(conf=conf or SparkConf())\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 144, in __init__\n SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 339, in _ensure_initialized\n SparkContext._gateway = gateway or launch_gateway(conf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/java_gateway.py\", line 108, in launch_gateway\n raise RuntimeError(\"Java gateway process exited before sending its port number\")\n\nMessage:\n\n Java gateway process exited before sending its port number\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}
Ketan (kumare3)
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: pods "ahqnwbjz2mzwg5hjbxkq-n0-0-driver" is forbidden: User "system:serviceaccount:flytesnacks-development:demo" cannot watch resource "pods" in API group "" in the namespace "flytesnacks-development"
flytectl register examples -p flytesnacks -d development
Samhita Alla
Ketan (kumare3)