I am getting a SIGSEGV error when running a task remotely. I am not sure how to debug. I can't tell if the SIGSEGV is coming from flyte or the from the task itself

│ │ fast_execute_task_cmd                                                        │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ ❱ 517 │   subprocess.run(cmd, check=True)                                    │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ /usr/lib/python3.9/subprocess.py:528 in run                                  │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ ❱  528 │   │   │   raise CalledProcessError(retcode, process.args,           │                                                                                                                                                      │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                                                                                      │
│ CalledProcessError: Command '['pyflyte-execute', '--inputs',                                                                                                                                                                          │
│ '<gs://my-bucket/metadata/propeller/flytesnacks-devel>                                                                                                                                                      │
│ opment-ff652f36eba884a609c5/n4/data/inputs.pb', '--output-prefix',                                                                                                                                                                    │
│ '<gs://my-bucket/metadata/propeller/flytesnacks-devel>                                                                                                                                                      │
│ opment-ff652f36eba884a609c5/n4/data/0', '--raw-output-data-prefix',                                                                                                                                                                   │
│ '<gs://my-bucket/zj/ff652f36eba884a609c5-n4-0>',                                                                                                                                                          │
│ '--checkpoint-path',                                                                                                                                                                                                                  │
│ '<gs://my-bucket/zj/ff652f36eba884a609c5-n4-0/_flyt>                                                                                                                                                      │
│ echeckpoints', '--prev-checkpoint', '""', '--dynamic-addl-distro',                                                                                                                                                                    │
│ '<gs://my-bucket/flytesnacks/development/5XLFFPD34ANQ>                                                                                                                                                      │
│ OZQYK5Z4AXNLBE======/script_mode.tar.gz', '--dynamic-dest-dir', '.',                                                                                                                                                                  │
│ '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--',                                                                                                                                                      │
│ 'task-module', 'mlops_prototype.workflows.evaluation_pipeline', 'task-name',                                                                                                                                                          │
│ 'evaluate_gpu']' died with <Signals.SIGSEGV: 11>.                                                                                                                                                                                     │
│

Wow what, this seems like a driver load failure

did you see any other error message in the pod or flyteconsole?

did not see any other errors.

are you able to share the code snippet

no unfortunately. I am going to try FlyteInteractive

@Alex Beach does FlyteInteractive work for your use case?

i was not able to get FlyteInteractive to work and had to move on to other things