https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Alex Beach

03/23/2024, 12:10 AM
I am getting a SIGSEGV error when running a task remotely. I am not sure how to debug. I can't tell if the SIGSEGV is coming from flyte or the from the task itself
Copy code
│ │ fast_execute_task_cmd                                                        │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ ❱ 517 │   subprocess.run(cmd, check=True)                                    │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ /usr/lib/python3.9/subprocess.py:528 in run                                  │                                                                                                                                                      │
│ │                                                                              │                                                                                                                                                      │
│ │ ❱  528 │   │   │   raise CalledProcessError(retcode, process.args,           │                                                                                                                                                      │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                                                                                      │
│ CalledProcessError: Command '['pyflyte-execute', '--inputs',                                                                                                                                                                          │
│ '<gs://my-bucket/metadata/propeller/flytesnacks-devel>                                                                                                                                                      │
│ opment-ff652f36eba884a609c5/n4/data/inputs.pb', '--output-prefix',                                                                                                                                                                    │
│ '<gs://my-bucket/metadata/propeller/flytesnacks-devel>                                                                                                                                                      │
│ opment-ff652f36eba884a609c5/n4/data/0', '--raw-output-data-prefix',                                                                                                                                                                   │
│ '<gs://my-bucket/zj/ff652f36eba884a609c5-n4-0>',                                                                                                                                                          │
│ '--checkpoint-path',                                                                                                                                                                                                                  │
│ '<gs://my-bucket/zj/ff652f36eba884a609c5-n4-0/_flyt>                                                                                                                                                      │
│ echeckpoints', '--prev-checkpoint', '""', '--dynamic-addl-distro',                                                                                                                                                                    │
│ '<gs://my-bucket/flytesnacks/development/5XLFFPD34ANQ>                                                                                                                                                      │
│ OZQYK5Z4AXNLBE======/script_mode.tar.gz', '--dynamic-dest-dir', '.',                                                                                                                                                                  │
│ '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--',                                                                                                                                                      │
│ 'task-module', 'mlops_prototype.workflows.evaluation_pipeline', 'task-name',                                                                                                                                                          │
│ 'evaluate_gpu']' died with <Signals.SIGSEGV: 11>.                                                                                                                                                                                     │
│
k

Ketan (kumare3)

03/24/2024, 3:40 PM
Wow what, this seems like a driver load failure
k

Kevin Su

03/25/2024, 3:46 PM
did you see any other error message in the pod or flyteconsole?
you could use FlyteInteractive to debug it
a

Alex Beach

03/25/2024, 4:47 PM
did not see any other errors.
k

Kevin Su

03/25/2024, 4:54 PM
are you able to share the code snippet
a

Alex Beach

03/25/2024, 4:55 PM
no unfortunately. I am going to try FlyteInteractive
d

David Espejo (he/him)

03/27/2024, 5:05 PM
@Alex Beach does FlyteInteractive work for your use case?
a

Alex Beach

03/27/2024, 5:14 PM
i was not able to get FlyteInteractive to work and had to move on to other things