https://flyte.org
Join Slack
What’s the minimum python version officially supported for flytekit? Was looking at a mypy improveme...
# flyte-support
e
What’s the minimum python version officially supported for flytekit? Was looking at a mypy improvement to allow the type checker to catch obvious errors in workflows but looks like it’ll need ParamSpec in order to correctly propagate the type annotations in the task/workflow/dynamic decorators. It’s part of Python 3.10, but is also available in typing_extensions for >= py3.7. Is it safe to assume that python 3.7 is the oldest version that flytekit should be guaranteed to work on?
t
Flytekit works on python 3.7. But the support for 3.7 ends in ~5 months, and it may not be guaranteed then.
👍 1
e
Great, this enhancement should work with typing extensions then.
@high-accountant-32689 I'd flagged a mypy issue a few weeks ago, it looks like https://peps.python.org/pep-0612/ provides the solution
This may require a larger refactor to how the task/workflow/dynamic decorators work though, but curious on your thoughts here (also cc @freezing-airport-6809 /@thankful-minister-83577)
t
what sort of refactor?
the inability of those decorators (and launch plans) to supply proper type hints has long been irking me. would be fantastic if you could help fix!
e
The fact that 
@task
& 
@task()
are both valid usages is where the trickiness comes in I believe. If we'd be willing to relax that and enforce 
@task()
as the correct usage, I think this is achievable Also not sure about the best way to wire the `P.args`/`P.kwargs` stuff into 
PythonFunctionTask
(
PythonFunctionTask
itself doesn't even appear to be callable/I think there are some other things that aren't quite right in the typing)
This is the minimal example where I'd like mypy to flag an error:
Copy code
from flytekit.core.task import task
from flytekit.core.workflow import workflow


@task()
def test_task(a: int, b: float) -> float:
    return a + b


@workflow()
def test_wf() -> float:
    return test_task(a=1.0, b="foo")
h
@elegant-australia-91422, I started working on this (bear in mind it's very alpha - only handles 
@task
, etc), but it's a start: https://github.com/flyteorg/flytekit/pull/1372/. I double-checked that this is supported in 
typing_extensions
(which also works in python 3.7 as per https://github.com/python/typing_extensions/commit/b697a12f2793655db99dde660bb47458ad36aa55), although I haven't pushed a commit that imports from there instead of 
typing
.
e
interesting..I think you may need to propagate `P.args`/`P.kwargs` for mypy to actually flag the error, was trying this on my own branch this morning and couldn't get mypy to fail w/ the above example
h
with the code as it sits right now in that PR I get this error:
Copy code
❯ mypy workflows/example2.py
workflows/example2.py:5: error: Missing positional argument "_task_function" in call to "task"  [call-arg]
workflows/example2.py:5: error: Argument 1 has incompatible type "Callable[[int, float], float]"; expected <nothing>  [arg-type]
Found 2 errors in 1 file (checked 1 source file)
e
I think that was related to my point above - I think the optional callable for the task function is only due to the fact that the decorator supports @task and @task(). By standardizing on the latter I believe you can avoid the branch
h
got it. This is going to break a lot of people if we go that route though...
e
Yeah, either it’s an api break or we introduce new decorators which are more opinionated
And then slowly phase out the old ones when flytekit has a major version bump?
t
i don’t think there will be a major version bump for a while fbofw. i think that’s at least a couple years away.
what’s the issue?
task and task() can’t be aligned?
👍 1
e
Yep, its effectively two different styles of decorators (and ParamSpec supports only the latter IIUC)
We may go ahead and write our own decorators internally in our codebase so we get the typing benefits
t
oh wait.. we can’t fix just one of them?
e
Could be...I can try sketching out a self-contained example with both styles of decorator & 
ParamSpec
and add that to the discussion on @high-accountant-32689’s PR. We're internally pushing to improve a lot of our typing (and where possible upstream improvements to 3rd-party libs) since it's better for the iteration loop to be able to catch these obvious issues at build time
👍 1
159 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by