Hey all, working on getting the Spark plugin installed on our Flyte cluster.. I followed the docs here https://docs.flyte.org/en/latest/deployment/plugin_setup/k8s/index.html to deploy and custom docker image similar to https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html The plugin example workflows succeed successfully, but custom workflows are failing with

22/11/09 20:41:34 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/11/09 20:41:34 INFO SharedState: Warehouse path is 'file:/root/spark-warehouse'.
{"asctime": "2022-11-09 20:41:35,835", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-11-09 20:41:35,835", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py\", line 472, in dispatch_execute\n        native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/core/type_engine.py\", line 800, in literal_map_to_kwargs\n        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/core/type_engine.py\", line 800, in <dictcomp>\n        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/core/type_engine.py\", line 764, in to_python_value\n        return transformer.to_python_value(ctx, lv, expected_python_type)\n      File \"/usr/local/lib/python3.10/site-packages/flytekit/types/pickle/pickle.py\", line 59, in to_python_value\n        with open(uri, \"rb\") as infile:\n\nMessage:\n\n    [Errno 2] No such file or directory: '/var/folders/n9/l_t6ghp503g_c3btld_vtrj80000gq/T/flyte-0u_yn2wu/raw/f307bbecb7ed78ceadfe26e17ce70f15/a9e378697a49254e4539ea4f69d37f2c'\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-11-09 20:41:35,835", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}

Any pointers on what is causing the error?

mind sharing the code of custom workflow?

@Edgar Trujillo did you manage to figure this one out on your own? if you did, mind sharing your solution here?