curved-kilobyte-59985
03/17/2024, 7:17 AM[1/1] currentAttempt done. Last Error: USER::
[f2975714e0c1349cf849-n0-0] terminated with exit code (1). Reason [Error]. Message:
│
│ ❱ 760 │ │ │ │ return __callback(*args, **kwargs) │
│ │
│ /usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py:508 in │
│ fast_execute_task_cmd │
│ │
│ ❱ 508 │ subprocess.run(cmd, check=True) │
│ │
│ /usr/local/lib/python3.8/subprocess.py:516 in run │
│ │
│ ❱ 516 │ │ │ raise CalledProcessError(retcode, process.args, │
╰──────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['pyflyte-execute', '--inputs',
'<gs://flyte-gcp-data-174222667125/metadata/propeller/flytesnacks-development-f29>
75714e0c1349cf849/n0/data/inputs.pb', '--output-prefix',
'<gs://flyte-gcp-data-174222667125/metadata/propeller/flytesnacks-development-f29>
75714e0c1349cf849/n0/data/0', '--raw-output-data-prefix',
'<gs://flyte-gcp-data-174222667125/0r/f2975714e0c1349cf849-n0-0>',
'--checkpoint-path',
'<gs://flyte-gcp-data-174222667125/0r/f2975714e0c1349cf849-n0-0/_flytecheckpoints>
', '--prev-checkpoint', '""', '--dynamic-addl-distro',
'<gs://flyte-gcp-data-174222667125/flytesnacks/development/VTSSZ7BWFXUL52RFOVN2A2>
2UCA======/script_mode.tar.gz', '--dynamic-dest-dir', '.', '--resolver',
'flytekit.core.python_auto_container.default_task_resolver', '--',
'task-module', 'experiments.workflows.workflow', 'task-name',
'preprocess_data']' returned non-zero exit status 1.
when i run this command poetry run pyflyte run --remote -d development experiments/workflows/workflow.py train_workflow
for a poetry project of this code structure
├── README.md
├── config.yaml
├── experiments
│ ├── __init__.py
│ ├── configs
│ │ ├── __init__.py
│ │ ├── config.py
│ │ └── training.yaml
│ ├── utils
│ │ ├── __init__.py
│ │ ├── loading.py
│ │ └── training.py
│ └── workflows
│ ├── __init__.py
│ └── workflow.py
├── gcp-artifact-reader.key
├── gcp-artifact-writer.key
├── pyproject.toml
├── tests
│ ├── __init__.py
│ └── test_workflow.py
└── tox.ini
tall-lock-23197
kubectl logs <pod-name> -n flytesnacks-development
curved-kilobyte-59985
03/18/2024, 10:48 AM