damp-lion-88352
07/31/2023, 5:05 AMFROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
RUN pip install -U git+<https://github.com/future-outlier/flytekit.git@master>
RUN pip install flytekitplugins-spark
RUN pip install marshmallow-enum
However, I face an error that I need to setup my java in my docker file on my flyte console
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/flytekit/core/base_task.py", line 511, in dispatch_execute
new_user_params = self.pre_execute(ctx.user_space_params)
File "/usr/local/lib/python3.9/site-packages/flytekitplugins/spark/task.py", line 168, in pre_execute
self.sess = sess_builder.getOrCreate()
File "/usr/local/lib/python3.9/site-packages/pyspark/sql/session.py", line 477, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 512, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 198, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 432, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/usr/local/lib/python3.9/site-packages/pyspark/java_gateway.py", line 106, in launch_gateway
raise RuntimeError("Java gateway process exited before sending its port number")
Message:
Java gateway process exited before sending its port number
SYSTEM ERROR! Contact platform administrators.
Is this because I need to use the docker file for Databricks mentioned in here?
https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/databricks_plugin/index.html#how-to-build-your-dockerfile-for-spark-on-databricks
But this doesn't look like a complete Docker file to use because I didn't install flytekit in the dockerfile
Can anyone clarify the concepts to use the plugins in pyflyte remote ?
Thanks a lotglamorous-carpet-83516
07/31/2023, 4:06 PMdamp-lion-88352
08/01/2023, 1:21 AM