millions-librarian-58501
01/26/2023, 6:35 AMfailed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute
│ handle for plugin [sidecar]: [BadTaskSpecification] invalid TaskSpecification [<nil>], Err: [nil Struct Object passed]
How can I handle it?millions-librarian-58501
01/26/2023, 6:35 AMtrain = ShellTask(
⚠ name="train",
debug=True,
script="""
set -ex
NPROC_PER_NODE=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
python -m torch.distributed.launch --nproc_per_node $NPROC_PER_NODE pocs/wav2vec2/flyte.py --cfg {inp>
""",
inputs=kwtypes(cfg=str, save_directory=str),
task_config=Pod(pod_spec=generate_gpu_pod_spec_for_task(), primary_container_name="primary"),
)
⚠ @workflow
def wf(cfg: str, save_directory: str) -> None:
train(cfg=cfg, save_directory=save_directory)
my workflowmillions-librarian-58501
01/26/2023, 6:36 AMmillions-librarian-58501
01/26/2023, 8:52 AM@task(task_config=Pod(…))
annotation, task’s config has k8sPod
item, however, shelltask with pod task_config, task’s config has container
item, not k8sPod
quaint-diamond-37493
01/26/2023, 11:33 AMhallowed-mouse-14616
01/26/2023, 2:38 PMPod
task_config
is only supported in python task definitions. There has been some great conversation on the restrictions on this and how we might address it.
Basically, we decided to support a pod_template
argument for all task definitions that will work exactly as you expected in your example above. We are tracking progress in this issue. The backend changes are close to being complete (hopefully this week) and then after completing the flytekit and testing end 2 end we will make this support available!millions-librarian-58501
01/26/2023, 2:55 PM