https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Frank Shen

11/29/2023, 2:59 AM
hello, I am trying to use spark-operator with flyte. However, I cannot pull the image.
Copy code
docker pull gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
Error response from daemon: manifest for gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0 not found: manifest unknown: Failed to fetch "v1beta2-1.2.0-3.0.0" from request "/v2/spark-operator/spark-operator/manifests/v1beta2-1.2.0-3.0.0".
(base) ➜  mlforge git:(flyte-deployment-2) ✗ docker pull gcr.io/spark-operator/spark-operator
Using default tag: latest
Error response from daemon: manifest for gcr.io/spark-operator/spark-operator:latest not found: manifest unknown: Failed to fetch "latest" from request "/v2/spark-operator/spark-operator/manifests/latest".
I got the image repo url from https://github.com/helm/charts/blob/master/incubator/sparkoperator/values.yaml, and image tag from https://googlecloudplatform.github.io/spark-on-k8s-operator/ Could anyone provide a working image tag? Thanks!
s

Samhita Alla

11/29/2023, 9:37 AM
@Frank Shen are you following the steps outlined in https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html doc?
h

Haytham Abuelfutuh

11/29/2023, 7:00 PM
If you have run it before, can you update your repo?
Copy code
helm repo update
I think the latest version is:
v1beta2-1.3.8-3.1.1
.. can you give this a shot? from here: https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml
f

Frank Shen

11/29/2023, 7:20 PM
Hi @Haytham Abuelfutuh, The same error:
Copy code
docker pull <http://gcr.io/spark-operator/spark-operator:v1beta2-1.3.8-3.1.1|gcr.io/spark-operator/spark-operator:v1beta2-1.3.8-3.1.1>
Error response from daemon: manifest for <http://gcr.io/spark-operator/spark-operator:v1beta2-1.3.8-3.1.1|gcr.io/spark-operator/spark-operator:v1beta2-1.3.8-3.1.1> not found: manifest unknown: Failed to fetch "v1beta2-1.3.8-3.1.1" from request "/v2/spark-operator/spark-operator/manifests/v1beta2-1.3.8-3.1.1".
Hi @Samhita Alla, I did. The issue is the image doesn’t exist. See above docker pull command I ran from local PC.
or this:
Copy code
docker pull <http://gcr.io/spark-operator/spark-operator|gcr.io/spark-operator/spark-operator>
Using default tag: latest
Error response from daemon: manifest for <http://gcr.io/spark-operator/spark-operator:latest|gcr.io/spark-operator/spark-operator:latest> not found: manifest unknown: Failed to fetch "latest" from request "/v2/spark-operator/spark-operator/manifests/latest".
CC: @Cyrus
h

Haytham Abuelfutuh

11/29/2023, 9:00 PM
@Frank Shen the image should be:
<http://ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.8-3.1.1|ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.8-3.1.1>
I wonder why is it your installation is trying to install from a non-existing repo... or maybe that was a more recent change from ghcr to gcr?
f

Frank Shen

11/29/2023, 9:06 PM
@Haytham Abuelfutuh, The spark-operator helm-chart is automatically using gcr from it’s templates which is wrong! I had to override to ghcr.io in my chart.
Thanks!
CC @Cyrus