https://flyte.org logo
#ask-the-community
Title
# ask-the-community
d

Dhruv Malik

01/15/2024, 5:42 PM
Hi i had query regarding how we can create a queue of the bacalhau tasks, so that in the workflow we can choose asynchronously which of the tasks are to be executed or others are to be pre-empted ? thanks
k

Ketan (kumare3)

01/15/2024, 5:49 PM
Is this a bacalhau question?
d

David Espejo (he/him)

01/16/2024, 2:25 PM
Cc @David Aronchick
d

David Aronchick

01/16/2024, 3:00 PM
Heya! Happy to help
Are you trying to create the queue in Bacalhau or in airflow?
d

Dhruv Malik

01/17/2024, 1:19 PM
Hi David (sorry i kind of mixed up the slack channels as initially i would've contacted bacalhau-general ) . i wanted to create queue of the various jobSpecs that are given on top of the bacalhau So i wanted to essentially is to implement a queue of the bacalhau jobs (json provided in the array) which create
BacalhauTask
objects and then puts them into queue, once we get the array of json_spec , we'll pop the task and then pass the correponding JobSpec to the given object and execute the job tasks. here is the gist of the pseudocode: https://gist.github.com/dhruvmalik007/17c1b644f799f4c2fe9130d217e9c67b . so basically being a automated version of the chained jobs: https://github.com/bacalhau-project/bacalhau/blob/main/integration/flyte/plugins/flytekit-bacalhau/examples/chained-jobs.py .
d

David Aronchick

01/17/2024, 7:33 PM
Oh interesting! So I guess this really is an airflow thing? Like would you want to create other types of jobs in a queue?
d

Dhruv Malik

01/18/2024, 7:02 AM
thanks @David Aronchick for response. so the fact that i am not well verse about Airflow but in my understanding i thought that flyte and Airflow are nearly same (DAG frameworks that allow scheduling the job) and that me and my collague decided to use flyte as the normal choice
d

David Aronchick

01/18/2024, 3:33 PM
Thanks! Can we move this to a mail? I don’t want this to get lost- aronchick@expanso.io
d

Dhruv Malik

01/19/2024, 9:23 AM
thanks and sounds good.
Hi David i did mail on this just yesterday with my colleague in CC.