Hello, I am having trouble building a custom image that will execute pyspark tasks. ```Last Error: U...

salmon-refrigerator-32115

10/21/2022, 5:24 PM

Hello, I am having trouble building a custom image that will execute pyspark tasks.

Copy code

Last Error: USER::Spark Job Failed with Error: driver container failed with ExitCode: 127, Reason: ContainerCannotRun

I have the following in the requirements.txt file for spark related dependencies.

Copy code

flytekit>=0.20.0
flytekitplugins-snowflake
flytekitplugins-spark
scikit-learn
xgboost
unionml
SHAP
pandas
apache-beam
joblib
pyyaml==6.0
boto3==1.9.207
aws-secretsmanager-caching==1.1.1.5
aws-requests-auth==0.4.3

salmon-refrigerator-32115

10/21/2022, 5:45 PM

I found this https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#spark-docker-image

salmon-refrigerator-32115

10/21/2022, 5:46 PM

It has the Dockerfile. Could you tell me the repo that hosts the dockerfile? Could you tell me what spark related dependencies will be needed in the requirements.txt?

salmon-refrigerator-32115

10/21/2022, 5:47 PM

I am thinking the following. Am I correct?

Copy code

flytekit>=0.20.0
flytekitplugins-spark

high-accountant-32689

10/21/2022, 8:51 PM

@salmon-refrigerator-32115, https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/Dockerfile

👍 1

156 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.