If we have an already containerized Spark job we w...
# announcementsk
Katrina P
08/25/2022, 8:35 PMIf we have an already containerized Spark job we would like to run as a raw container using ContainerTask(), can we use this task in a workflow and ensure this gets submitted to the Spark Operator to run in cluster mode? cc: @Vasu Upadhya
k
Ketan (kumare3)
08/25/2022, 11:55 PM
It’s raw container - you have to do it
Ketan (kumare3)
08/25/2022, 11:55 PM
But I don’t understand completely- is this an independent spark job? Not using flytekit
k
Katrina P
08/26/2022, 2:39 PMYes, it is an independent container running a spark job; we want to use some existing jobs we have running outside of flyte in our workflow without having to re-write the jobs for now (if possible)
k
Ketan (kumare3)
08/26/2022, 2:43 PM
Spark on k8s needs some specif container
Ketan (kumare3)
08/26/2022, 2:43 PM
once you have that absolutely you can use raw container
Ketan (kumare3)
08/26/2022, 2:44 PM
woth right service account
k
Katrina P
08/26/2022, 3:47 PMYes the container runs in spark already with the SparkOperator outside of flyte
Katrina P
08/26/2022, 3:47 PMWe just want to be able to run it as a task as part of a workflow
k
Ketan (kumare3)
08/26/2022, 3:47 PM
Ya that should be fine
k
Katrina P
08/26/2022, 3:48 PMWould Flyte submit it to the SparkOperator or would it end up being run in single node mode?
Katrina P
08/26/2022, 3:48 PM(assuming we use spark service acount to run the workflow)
k
Ketan (kumare3)
08/26/2022, 3:51 PM
Ohh Flyte will not submit
To spark operator
Ketan (kumare3)
08/26/2022, 3:51 PM
i thought you want in client mode
Ketan (kumare3)
08/26/2022, 3:51 PM
flyte will only submit if task type is spark
Ketan (kumare3)
08/26/2022, 3:51 PM
and some fields are
Set, check the flytekit plugin for more details
k
Katrina P
08/26/2022, 3:52 PMgot it, thanks
158 Views