Hello, Regarding I found this https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#spark-docker-image, and its example Dockerfile.

RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

RUN pip3 install wheel

# Install Python dependencies
COPY k8s_spark/requirements.txt /root
RUN pip install -r /root/requirements.txt

RUN flytekit_install_spark3.sh

you could follow this guide to setup the cluster and write a docker image for spark job.

@glamorous-carpet-83516, thank you. That’s exactly what I was looking for.