Fredrik Lyford
12/30/2022, 6:55 AMSamhita Alla
Ketan (kumare3)
Samhita Alla
but it would be really nice to be able to take start date and number of days back as an input to the workflow (the blocker is that we cannot iterate over a Promise).You can use a dynamic workflow to perform iterations over workflow inputs. (thanks @Dan Rammer (hamersaw))
Ketan (kumare3)
Fredrik Lyford
01/02/2023, 7:36 AMKetan (kumare3)
Niels Bantilan
01/03/2023, 3:17 PMbut it would be really nice to be able to take start date and number of days back as an input to the workflowIf I understand correctly, is this what you want to achieve at a high level? Assumptions: • I have a
process_dataset
routine that processes a single dataset for a particular date
• Every day I process today’s dataset, in addition to datasets for some number of days back.
• Each invocation of process_dataset
for a particular day’s dataset is completely independent from processing other day’s datasets.
Every day, kick off scheduled job at `kickoff_datetime`:
1. Pass in kickoff_datetime
into main entrypoint, in addition to a num_days_back
parameter
2. Generate a list of dates, which includes today’s date and some num_days_back
worth of days
3. Apply process_dataset
to each dataset in an embarrassingly parallel fashion
Does this seem correct?Ketan (kumare3)
Fredrik Lyford
01/04/2023, 11:10 AMNiels Bantilan
01/05/2023, 2:54 PMFredrik Lyford
01/05/2023, 3:13 PMKetan (kumare3)
Fredrik Lyford
01/06/2023, 8:17 AMKetan (kumare3)