Hello, I am having trouble building a custom image...
# ask-the-community
f
Hello, I am having trouble building a custom image that will execute pyspark tasks.
Copy code
Last Error: USER::Spark Job Failed with Error: driver container failed with ExitCode: 127, Reason: ContainerCannotRun
I have the following in the requirements.txt file for spark related dependencies.
Copy code
flytekit>=0.20.0
flytekitplugins-snowflake
flytekitplugins-spark
scikit-learn
xgboost
unionml
SHAP
pandas
apache-beam
joblib
pyyaml==6.0
boto3==1.9.207
aws-secretsmanager-caching==1.1.1.5
aws-requests-auth==0.4.3
It has the Dockerfile. Could you tell me the repo that hosts the dockerfile? Could you tell me what spark related dependencies will be needed in the requirements.txt?
I am thinking the following. Am I correct?
Copy code
flytekit>=0.20.0
flytekitplugins-spark
157 Views