Nimrod Rak09/05/2022, 6:59 AM
Shivay Lamba09/05/2022, 7:20 AM
Nimrod Rak09/05/2022, 7:31 AM
Shivay Lamba09/05/2022, 7:37 AM
Kevin Su09/05/2022, 8:16 AM
We have a
@task( requests=Resources( gpu=gpu, mem=mem, storage=storage, ephemeral_storage=ephemeral_storage ), ) def pytorch_mnist_task(): ...
transformer in flytekit, which means you can directly return a
in your task. Flyte will help your save this data in s3, and the downstream task can fetch data from the bucket.
@task def t1() -> torch.Tensor: return torch.tensor([[1.0, -1.0, 2], [1.0, -1.0, 9], [0, 7.0, 3]]) @task def t2(df: torch.Tensor) ... @workflow def wf(): df = t1() t2(df=df)
Nimrod Rak09/05/2022, 8:20 AM
Kevin Su09/05/2022, 8:20 AM
Nimrod Rak09/05/2022, 8:21 AM
Kevin Su09/05/2022, 8:25 AM
Nimrod Rak09/05/2022, 8:35 AM
Great to hear about the parallelization, so there is a dispatcher that determines what receives what resources etc?Yes! Refer to https://docs.flyte.org/en/latest/deployment/cluster_config/performance.html#worst-case-workflows-poison-pills-max-parallelism for an in-depth view of workers to run the flyte code.
I don’t want copies because all I need is a pipeline to connect units that already run on GPU and the copies are just overhead.Copying is inevitable cause the data would be uploaded and downloaded in case of tasks’ communication. If there’s no copy available, you wouldn’t be able to access the tensor in your other task where it’s an input.
Nimrod Rak09/05/2022, 9:48 AM