Seung-Woo Lee
01/26/2023, 6:35 AMfailed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute
│ handle for plugin [sidecar]: [BadTaskSpecification] invalid TaskSpecification [<nil>], Err: [nil Struct Object passed]
How can I handle it?train = ShellTask(
⚠ name="train",
debug=True,
script="""
set -ex
NPROC_PER_NODE=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
python -m torch.distributed.launch --nproc_per_node $NPROC_PER_NODE pocs/wav2vec2/flyte.py --cfg {inp>
""",
inputs=kwtypes(cfg=str, save_directory=str),
task_config=Pod(pod_spec=generate_gpu_pod_spec_for_task(), primary_container_name="primary"),
)
⚠ @workflow
def wf(cfg: str, save_directory: str) -> None:
train(cfg=cfg, save_directory=save_directory)
my workflow@task(task_config=Pod(…))
annotation, task’s config has k8sPod
item, however, shelltask with pod task_config, task’s config has container
item, not k8sPod
Felix Ruess
01/26/2023, 11:33 AMDan Rammer (hamersaw)
01/26/2023, 2:38 PMPod
task_config
is only supported in python task definitions. There has been some great conversation on the restrictions on this and how we might address it.
Basically, we decided to support a pod_template
argument for all task definitions that will work exactly as you expected in your example above. We are tracking progress in this issue. The backend changes are close to being complete (hopefully this week) and then after completing the flytekit and testing end 2 end we will make this support available!Seung-Woo Lee
01/26/2023, 2:55 PM