https://flyte.org logo
Join Slack
Powered by
<#1556 General Partial support in flytekit and mul...
# flyte-github
a
#1556 General Partial support in flytekit and multi-list support in flytekit Pull request opened by kumare3 TL;DR This Pull Requests adds few very higly requested features. Following are the examples of Tasks and workflows that will be supported after this PR Support for partially binding values to a map task. This will reduce the need to use 
dataclasses
as the only way of passing multiple values to a map task.
Copy code
from flytekit import task, workflow, map_task
import typing
import functools

@task
def generate(n: int) -> typing.List[int]:
   return list(range(n))

@task
def square_add(i: int, j: int) -> int:
   return i*i + j

@workflow
def wf(n: int) -> typing.List[int]:
   l = generate(n=n)
   square = functools.partial(square_add, j=10)
   return map_task(square)(i=l)
It is possible to bind promises to partials
Copy code
@workflow
def wf(n: int) -> typing.List[int]:
   l = generate(n=n)
   square = functools.partial(square_add, j=n). # notice n is a promise
   return map_task(square)(i=l)
*Support for passing multiple lists using map_tasks *
Copy code
from flytekit import task, workflow, map_task
import typing
import functools

@task
def generate(n: int) -> typing.List[int]:
   return list(range(n))

@task
def square_add(i: int, j: int) -> int:
   return i*i + j

@workflow
def wf(n: int) -> typing.List[int]:
   l = generate(n=n)
   return map_task(p)(i=l, j=l)
Copy code
from flytekit import task, workflow, map_task
import typing
import functools

@task
def generate(n: int) -> typing.List[int]:
   return list(range(n))

@task
def square_add(i: int, j: int, k: int) -> int:
   return i*i + j + k

@workflow
def wf(n: int) -> typing.List[int]:
   l = generate(n=n)
   p = functools.partial(square_add, k=n)
   return map_task(p)(i=l, j=l)
Note: In addition this PR also verifies that it is possible to use partial functions in flytekit, as follows,
Copy code
from flytekit import task, workflow
import functools

@task
def tk(i: int) -> int:
   return i + 1

@workflow
def wf():
  p = functools.partial(tk, i=10)  # Note binding to a constant value
  x =  p()
  p2 = functools.partial(tk, i=x)  # Note binding to a promise
  p2()
Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#3403 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks
Open in Slack
PreviousNext
Powered by