Hi. I have yet another problem running the example...
# ask-the-communitył
Łukasz Trojanowski
09/05/2023, 3:10 PMHi. I have yet another problem running the example task from the spark task docs https://docs.flyte.org/projects/cookbook/en/stable/auto_examples/k8s_spark_plugin/index.html.
When running
pyflyte run --remote pyspark_pi.py my_spark Copy code
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flytekit/exceptions/scopes.py", line 206, in user_entry_point
return wrapped(*args, **kwargs)
File "/root/pyspark_pi.py", line 51, in hello_spark
count = sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
File "/usr/local/lib/python3.9/dist-packages/pyspark/rdd.py", line 1905, in reduce
vals = self.mapPartitions(func).collect()
File "/usr/local/lib/python3.9/dist-packages/pyspark/rdd.py", line 1814, in collect
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/usr/local/lib/python3.9/dist-packages/pyspark/rdd.py", line 5441, in _jrdd
wrapped_func = _wrap_function(
File "/usr/local/lib/python3.9/dist-packages/pyspark/rdd.py", line 5243, in _wrap_function
return sc._jvm.SimplePythonFunction(
Message:
'JavaPackage' object is not callable
User error.s
Samhita Alla
09/06/2023, 7:25 AM
Assuming you're using ImageSpec, is the image being pushed to the registry you mentioned?
cc @Kevin Su
ł
Łukasz Trojanowski
09/06/2023, 12:49 PMYes it did get pushed https://hub.docker.com/r/ltrojanowski/flyte-spark-plugin
k
Kevin Su
09/06/2023, 5:12 PMcould you try pyflyte register --no-fast …, and run it again
s
Samhita Alla
09/07/2023, 5:08 AM
pyflyte register --non-fast <package-or-module>3 Views