Frank Shen
01/07/2023, 12:43 AMpy4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.IOException: Failed to connect to <http://nyxmmedina01741.wmad.warnermedia.com/10.217.173.85:55808|nyxmmedina01741.wmad.warnermedia.com/10.217.173.85:55808>
Ketan (kumare3)
Frank Shen
01/07/2023, 12:47 AMYee
Ketan (kumare3)
Frank Shen
01/07/2023, 1:01 AMconf = SparkConf()
flyte manages spark context differently.
flytekit.current_context().spark_session
Ketan (kumare3)
Frank Shen
01/07/2023, 1:21 AMflytekit.current_context()
Yee
user_params.builder().add_attr
is what makes it available in the object returned by current_context()Samhita Alla
export SPARK_LOCAL_IP="127.0.0.1"
Frank Shen
01/09/2023, 6:24 PMadmin:
endpoint: dns:///flyte.dev.dap.warnermedia.com
SPARK_LOCAL_IP:
127.0.0.1
@task(
task_config=Spark(
spark_conf={
...
# The following is needed only when running spark task in dev's local PC. Also need to do this locally: export SPARK_LOCAL_IP="127.0.0.1"
"spark.jars.packages": "org.apache.hadoop:hadoop-aws:3.3.2",
"spark.hadoop.fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider",
"spark.hadoop.fs.s3a.access.key": "",
"spark.hadoop.fs.s3a.secret.key": "",
"spark.hadoop.fs.s3a.session.token": "",
},
Kevin Su
01/10/2023, 12:46 AMSamhita Alla
can I add SPARK_LOCAL_IP to .flyte/config.yaml?I don't think so. You can, however, add it to the bash or zsh profile since it's an env variable.
Frank Shen
01/10/2023, 6:23 PMKevin Su
01/10/2023, 6:26 PMFrank Shen
01/10/2023, 6:26 PMKevin Su
01/10/2023, 6:28 PMspark-defaults.conf
, and add it to env. pyspark will use default config in it.
https://stackoverflow.com/a/71214326/9574775Frank Shen
01/10/2023, 6:34 PMThe spark-defaults.conf file should be located in:
$SPARK_HOME/conf
Kevin Su
01/10/2023, 6:40 PMFrank Shen
01/10/2023, 6:46 PMKevin Su
01/10/2023, 6:49 PMFrank Shen
01/10/2023, 6:49 PMKetan (kumare3)
Frank Shen
01/11/2023, 5:37 PMKevin Su
01/11/2023, 6:45 PMFrank Shen
01/11/2023, 6:59 PMYee
Frank Shen
01/11/2023, 7:41 PM