If we have an already containerized Spark job we would like

Question

If we have an already containerized Spark job we would like to run as a raw container using ContainerTask can we use this task in a workflow and ensure this gets submitted to the Spark Operator to run in cluster mode cc < Vasu Upadhya>

Ketan (kumare3) · Answer

It s raw container you have to do it

Ketan (kumare3) · Answer

But I don t understand completely is this an independent spark job Not using flytekit

Katrina P · Answer

Yes it is an independent container running a spark job we want to use some existing jobs we have running outside of flyte in our workflow without having to re write the jobs for now if possible

Ketan (kumare3) · Answer

Spark on k8s needs some specif container

Ketan (kumare3) · Answer

once you have that absolutely you can use raw container

Ketan (kumare3) · Answer

woth right service account

Katrina P · Answer

Yes the container runs in spark already with the SparkOperator outside of flyte

Katrina P · Answer

We just want to be able to run it as a task as part of a workflow

Ketan (kumare3) · Answer

Ya that should be fine

Katrina P · Answer

Would Flyte submit it to the SparkOperator or would it end up being run in single node mode

Katrina P · Answer

assuming we use spark service acount to run the workflow

Ketan (kumare3) · Answer

Ohh Flyte will not submit To spark operator

Ketan (kumare3) · Answer

i thought you want in client mode

Ketan (kumare3) · Answer

flyte will only submit if task type is spark

Ketan (kumare3) · Answer

and some fields are Set check the flytekit plugin for more details

Katrina P · Answer

got it thanks