acoustic-carpenter-78188
12/22/2023, 8:33 PM@workflow
def wf(a: int = 3):
...
The 3
here will get picked up as the default value in any launch plan that gets created from this workflow (unless overridden at launch plan creation time).
This also works today:
@workflow
def wf(a: List[int] = [1, 2, 3]):
...
However, this is not Pythonic and may lead to errors. Specifically, this is actually an anti-pattern.
The correct way to handle complex default values in Python is to make the default None
and then check for that in the code, and then setting the input to the default in code. This isn't really doable in flytekit today (or possibly ever). Also, flytekit today doesn't handle None
values and there is no such thing as an Optional
type in Flyte IDL (yet, union types may be supported in the future).
This issue also affects maps, and all other mutable custom objects (protos or dataclass types).
First Need To Test
We should add some tests to flytekit first to see how big of a problem this is, and actual user-code-pattern that breaks. This should help refine the scope of this ticket, and should help inform a solution.
Options
Leave it as is
While it is an anti-pattern, it is concise and keeps the default-specification pattern consistent between primitives and more complex types. We can try to leave it as is, but add code to deep copy the values before passing them on. A deep copy already happens with lists and dicts and such because when a workflow is run locally, it already gets converted into Flyte Literals before tasks get a chance to mess with them.
Add options to the workflow decorator
We can specify things in the workflow decorator
@workflow(a=[1,2,3])
def wf(a: Optional[List[int]] = None):
...
We'll still need to do a deep copy, because we're still only given one copy of the literal. The type engine will have to understand that we're not really trying to specify an Optional type, and launch plans will pick up the default value when created.
Require frozen things
There's a frozen list (though no frozen dict) that we can maybe force people to use when declaring defaults. This adds complexity in the type engine however.
flyteorg/flyteacoustic-carpenter-78188
12/22/2023, 8:33 PM