https://flyte.org
Join Slack
Hi Team, We are trying to run our workflow in GPU, but when we are running the flyte task we are get...
# flyte-support
h
Hi Team, We are trying to run our workflow in GPU, but when we are running the flyte task we are getting following error:
Copy code
File "/opt/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1024, in _new_process_group_helper
        backend_class = ProcessGroupNCCL(backend_prefix_store, group_rank, group_size, pg_options)

Message:

    ProcessGroupNCCL is only supported with GPUs, no GPUs found!
We have also added the check in the start of the task about GPU availability and what i can see is that torch is able to identify the GPU :
Copy code
INFO:dummy.test_gpu:GPU is available in your machine with following device count : 1
2023-09-13 17:00:31,611 [dummy.test_gpu] [INFO ]  GPU is available in your machine with following device count : 1 
GPU available: True (cuda), used: True
The EC2 instance type we are using : 
g4dn.xlarge
Can anyone help me with this. Is it possible to define the instance type in flytekit.task resources
f
You can specify instance through labels
You shouldn’t do that inverse pattern
But coming soon to a Flyte near you us gpu type selection- cc @freezing-boots-56761
@helpful-church-28990 can you share your snippet of config in the task decorator
h
Copy code
@task(requests=Resources(cpu="100m", mem="1Gi", gpu="1"),
    limits=Resources(cpu="200m", mem="6Gi", gpu="1"),
)
def test_run(
Its a pretty simple config nothing special i have added
f
Ohh then why are you using distributed training?
You don’t need a process group
h
Sorry what do you mean by distributed training (I am new bie to this field)
f
If you want to still set it up - my recommendation use flytekit elastic plugin
Vipul - no worries- single gpu vs multi gpu training is different- from the error it seems your code is trying to setup multi gpu
h
Ok but in that case i dont think at this moment we need to use multi gpu. So as per my understanding you are suggesting to use to 
flytekit elastic plugin
to use only single Single gpu
f
It will scale from one to n
And I agree you shouldn’t need to use more than one
But then you have to modify your code
h
But then you have to modify your code
Thats fine we can do that. but would it be possible to share about how can i define the instance type in flyte recources section
t
You cannot specify GPU type in the task resources. That's still being worked on: https://github.com/flyteorg/flyte/discussions/3796. If you aren't using distributed training, not sure why your code is hitting that specific code. Are you running an example from the docs?
h
I am running the code provided by our Data Scientist, but i suspect is that i am using number of worker as 4 (but as a suspect)
t
Could you share your configuration?
h
Copy code
metadata:
    name: local
    phase: serve
  training:
    accelerator: gpu
    num_workers: 4
    hyperparameters:
      batch_size: 4096
      embedding_dim: 10
      weight_decay: 1e-6
      lr: 0.005
      optim_func: XXX
      loss_func: XXX
      max_epochs: 10
t
I meant, are you using any plugin?
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by