hi all 👋, I just wanted to ping the community to ask a quick question 🤔: Say you have workflow that uses a trained model to generate predictions on a scheduled launchplan. The question is, how do you typically want to get features for that prediction? I.e. when the scheduled workflow kicks off, where are you reading those features from? Do you need the kick-off time as a parameter to fetching data from, say, an s3 bucket or DB?

What I have seen in the past is a query against a metastore - or a datawarehouse

Do you need the kick-off time as a parameter to fetching data from, say, an s3 bucket or DB?

Cool, so that’s a “yes” to this question

Yep agreed that a date parameter is important. Whether we're training a model or batch inference, we'll always do it 'as of' some date. Something like Airflow's logical_date would be great.

Old thread but I want to chime in and say "yes absolutely". Tangential to this is being able to re-run a failed workflow at the originally scheduled time. I have not found a way of doing it but this is a crucial feature for e.g. backfilling jobs.

@worried-lighter-79998 this is something we are thinking of adding. Today you should be able to run as running is simply an adhic execution

How would you expose a time stamp and at the same time have it work with scheduling? If you set the time stamp parameter to datetime.now() in your launch plan it is executed at build time, not executive time.

No, Flyte allows timestamp to be variable for scheduled workflows right?

That has not been my experience

Would love to understand

Thank you for the link. So kickoff_time is a special arg which gets supplied by the scheduler? Is it exposed though the web ui as well?

Yes

Thank you very much. So to clarify the error I made: I had in the launch plan specified something like

default_inputs={"execution_time": datetime.now()}

, which is evaluated to the build time

Sorry for confusion @worried-lighter-79998

But if I understand you correctly i can remove that line and add

kickoff_time_input_arg="execution_time",

and it should be fine

Sadly this will be allowed as Flyte thinks you want a constant time of the build time as an arg

Yeah that makes sense in hindsight and was an error on my side

No docs should cater to avoiding confusion- please recommend an edit

If you think other people will make the same mistake it could be worth adding to the docs page something like "the scheduler specification and its arguments are executed when the flyte resources are compiled so something like

default_args={"kickoff_time": datetime.now()}

won't get the scheduled time but the build time. That's why we have

kickoff_time_input_arg="kickoff_time"

..."

Good idea