Do we have to change the default USER for spark tasks like above?

It's not mandatory but it's beneficial (and preferred) to set it to a non-root user.

@tall-lock-23197, @glamorous-carpet-83516, @thankful-minister-83577, Could you elaborate? I have a task that utilizes Ray running locally. And Ray needs to mkdir internally and it got denied when the task is run in spark SA. I am thinking to switch to default USR for the image that has spark capability to prevent endless permission issues like that.

File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 203, in user_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/dai_mle_models/xgboost/xgboost_tune_wf.py", line 83, in xgb_tune
        tuner = tune.Tuner(
      File "/opt/venv/lib/python3.9/site-packages/ray/tune/tuner.py", line 152, in __init__
        self._local_tuner = TunerInternal(**kwargs)
      File "/opt/venv/lib/python3.9/site-packages/ray/tune/impl/tuner_internal.py", line 104, in __init__
        self._experiment_checkpoint_dir = self._setup_create_experiment_checkpoint_dir(
      File "/opt/venv/lib/python3.9/site-packages/ray/tune/impl/tuner_internal.py", line 273, in _setup_create_experiment_checkpoint_dir
        os.makedirs(path, exist_ok=True)
      File "/usr/lib/python3.9/os.py", line 215, in makedirs
        makedirs(head, exist_ok=exist_ok)
      File "/usr/lib/python3.9/os.py", line 225, in makedirs
        mkdir(name, mode)

Message:

    [Errno 13] Permission denied: '/ray_results'

May I know why you are using spark service account to run Ray tasks?

@tall-lock-23197, @glamorous-carpet-83516, @thankful-minister-83577, Isn’t this a common scenario when you have non-spark (that uses Ray) and spark tasks co-exist in a workflow and they use the same docker image that is based of flyte spark?https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/Dockerfile? Since the Flyte Dockerfile has:

# Set /root user and group
# RUN chown -R ${spark_uid}:${spark_uid} /root
...
# USER ${spark_uid}

The best practice is using different images for those two tasks.

@glamorous-carpet-83516, thanks. However when I have non-spark and spark tasks in the same workflow file (with imports from both custom and spark dependencies), I still have to have both custom dependencies and spark dependencies built in to the same image. Therefore both images will be the same except one is using the root user and the other is using spark_uid user _*USER ${spark_uid}*_. What’s your comment on that?

one way to avoid import spark dependencies in the ray task is to import spark specific package in the @task, like

@task
def spark_task:
    import spark
    ...

@glamorous-carpet-83516, by spark task I meant pyspark task

@task
    task_config=Spark(
        spark_conf={
def a()

Yeah, it won't. You need to import the plugins globally. @glamorous-carpet-83516 @high-accountant-32689, I think we need to improve the UX here. If all the tasks are associated with custom images, we need to enable some sort of mechanism to simplify specifying the default image. Not sure if that's a possibility though. In this case, there needs to be a single image with user set to root or three images — a default image with the two plugins installed, one for Spark and one for Ray.

I used a single image with all dependencies combined, and with user set to root. It runs with spark SA and it runs workflow with spark and non-spark tasks (including Ray) fine.