GitHub
05/19/2023, 8:10 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytekit/commit/47be97412bdfd90e084dabb970ca587b202538ad|47be9741>
- Pass verify flag to all authenticators (#1641)
flyteorg/flytekitGitHub
05/19/2023, 8:47 PMfoo
is registered in fast mode, while bar
is registered in non-fast mode.
The subtask foo
is executed with pyflyte-execute
rather than pyflyte-fast-execute
when called from within bar
, causing ModuleNotFoundError
.
# a.py
@task
def foo(x: int) -> int:
return x
# This task foo was registered in fast registration
# b.py
@reference_task(
project="flytesnacks",
domain="development",
name="a.foo",
version="v0.1.0",
)
def foo(x: int) -> int:
...
@dynamic(image="...")
def bar(x: int) -> int:
for i in range(10):
foo(x)
# This task bar was registered in non-fast mode
Expected behavior
foo
should be executed with pyflyte-fast-execute
.
Additional context to reproduce
flytekit==1.4.2 and using flyte-binary
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/19/2023, 10:24 PMlicenses
script to package.json
. It prints a
Tracking Issue
_Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_
fixes https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flyteconsole
GitHub Actions: Build & Push FlyteConsole Image
GitHub Actions: Get Release Tag
GitHub Actions: Generate Release
WIP: Title contains "wip"
✅ 5 other checks have passed
5/9 successful checksGitHub
05/19/2023, 10:24 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/c69f010959a2d854bac7a694baf5e5a635aeae8b|c69f0109>
- Feature: Fullview Flyte Deck modal (#764)
flyteorg/flyteconsoleGitHub
05/19/2023, 10:24 PMGitHub
05/19/2023, 11:23 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/06fffc786edfe9d4cbb891d6c0e14181a5823998|06fffc78>
- Allow annotated FlyteFile as task input argument (#1632)
flyteorg/flytekitGitHub
05/20/2023, 12:48 AMs3fs
in the core image was outdated. We set a lower-bound in flyteorg/flytekit@4251751, but that only went out in flytekit 1.6.0, so let's use latest flytekit.
flyteorg/flytesnacks
GitHub Actions: Serialize & Register Flytesnacks workflow / Register example to sandbox
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/ray_example)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/pod)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kftensorflow)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfpytorch)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfmpi)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_spark)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_dask)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/whylogs_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/sql)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/papermilltasks)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/pandera_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/onnx_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/modin_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/mlflow_example)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/core)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/case_studies/ml_training/spark_horovod)
✅ 13 other checks have passed
13/30 successful checksGitHub
05/20/2023, 12:57 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytesnacks/commit/19a410752cd6bd9a41b6906656473e1da04ae0c4|19a41075>
- Use flytekit 1.6 in core image (#991)
flyteorg/flytesnacksGitHub
05/20/2023, 2:02 AMchain_entities.py
.
flyteorg/flytesnacks
GitHub Actions: Mark github pre-release as Release
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
✅ 26 other checks have passed
26/30 successful checksGitHub
05/20/2023, 5:28 AMGitHub
05/20/2023, 7:08 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytesnacks/commit/2c457dfec000912bb22e527fee6229c8808910c0|2c457dfe>
- Remove awscli from core and use botocore to create an s3 client (#992)
flyteorg/flytesnacksGitHub
05/20/2023, 8:18 AMGitHub
05/20/2023, 11:01 PMGitHub
05/20/2023, 11:32 PMwith_overrides
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
https://flyte-org.slack.com/archives/CP2HDHKE1/p1683934828365009
Follow-up issue
NA
flyteorg/flytekit
GitHub Actions: build (windows-latest, 3.8)
✅ 29 other checks have passed
29/30 successful checksGitHub
05/20/2023, 11:57 PMdynamic
workflows requires fully-specifying the arguments to a task
. This behavior is clunky in the context of map tasks, where users need to write a bunch of code to prepare the inputs to a map task (see here.
The purpose of this issue is to support partial tasks, which allow users to partially bind inputs to a task so that it can then be invoked with the remaining arguments, similar to https://docs.python.org/3/library/functools.html#functools.partial.
Goal: What should the final outcome look like, ideally?
Requirements
Partial tasks should:
1. support partial task inputs that are python values
2. support partial task inputs that are promises
3. support partial task definitions in the top-level scope of a module
4. support use of partial tasks in map tasks
Once partial tasks are implemented, it should look something like:
Support partial task inputs that are python values
from flytekit import task, workflow, partial
@task
def t1(x: int, y: float) -> float:
return x + y
@workflow
def wf(y: float):
partial_t1 = partial(t1, x=5)
return partial_t1(y=y)
Support partial task inputs that are promises
from flytekit import task, workflow, partial
@task
def t1(x: int, y: float) -> float:
return x + y
@task
def t2() -> int:
return 5
@workflow
def wf(y: float):
partial_t1 = partial(t1, x=t2())
return partial_t1(y=y)
Support partials in the top-level module scope
If a user wants to specify partial tasks using python literal values in the top-level scope:
from flytekit import task, workflow, partial
@task
def t1(x: int, y: float) -> float:
return x + y
partial_t1 = partial(t1, x=5)
@workflow
def wf(y: float):
return partial_t1(y=y)
Support partials in map tasks
from flytekit import task, workflow, partial, map_task
@task
def t1(x: int, y: float) -> float:
return x + y
@workflow
def wf(y: List[float]):
partial_t1 = partial(t1, x=5)
return map_task(partial_t1)(y=y)
Describe alternatives you've considered
Besides syntactic alternatives, not supporting partial tasks would require writing additional code to make some of Flyte's constructs work (e.g. map tasks with multiple inputs).
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/20/2023, 11:58 PMGitHub
05/21/2023, 1:08 AM$ flytectl demo start
$ docker restart flyte-sandbox
$ docker logs
...
{"json":{"routine":"worker-8","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
{"json":{"routine":"worker-7","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
{"json":{"routine":"worker-12","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
{"json":{"routine":"worker-9","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
{"json":{"routine":"worker-10","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
{"json":{"routine":"worker-11","src":"workers.go:113"},"level":"info","msg":"Started Worker","ts":"2022-09-05T11:49:35Z"}
2022/09/05 11:49:38 /go/pkg/mod/github.com/flyteorg/flyteadmin@v1.1.29/pkg/repositories/database.go:113
[error] failed to initialize database, got error failed to connect to `host=localhost user=postgres database=flyteadmin`: dial error (dial tcp [::1]:30089: connect: cannot assign requested address)
2022/09/05 11:49:38 /go/pkg/mod/github.com/flyteorg/datacatalog@v1.0.1/pkg/repositories/config/postgres.go:73
{"json":{"src":"initialize.go:22"},"level":"fatal","msg":"failed to connect to `host=localhost user=postgres database=flyteadmin`: dial error (dial tcp [::1]:30089: connect: cannot assign requested address)","ts":"2022-09-05T11:49:38Z"}
[error] failed to initialize database, got error failed to connect to `host=localhost user=postgres database=flyteadmin`: dial error (dial tcp [::1]:30089: connect: cannot assign requested address)
Screenshots
image▾
GitHub
05/21/2023, 3:59 AMGitHub
05/22/2023, 2:58 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/d7060cc13be711ea37773435abcc7fbca6c3e4df|d7060cc1>
- Use logger instead of print statement in sqlalchemy plugin (#1651)
flyteorg/flytekitGitHub
05/22/2023, 4:00 PMGitHub
05/22/2023, 4:04 PMGitHub
05/22/2023, 4:48 PMGitHub
05/22/2023, 4:58 PMGitHub
05/22/2023, 5:16 PMGitHub
05/22/2023, 5:26 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyteidl/commit/f091fd6da363cf9920701d29d221546478f787be|f091fd6d>
- Add namespace to execution system metadata (#406)
flyteorg/flyteidlGitHub
05/22/2023, 5:27 PMGitHub
05/22/2023, 5:28 PMGitHub
05/22/2023, 6:04 PMGitHub
05/22/2023, 6:14 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytesnacks/commit/2cc66f4b88444376bd0fdd4d07ed1efc43f49442|2cc66f4b>
- Rename session in chain entities (#993)
flyteorg/flytesnacksGitHub
05/22/2023, 6:45 PMmessageFormat
field expects a proto generated enum in the configuration. This doc requests a string value (ie. "unknown", "json", or "csv") which, if provided, will result in a failure to parse the configuration. This PR updates the docs to explain that messageFormat
is an optional parameter that is unused anywhere in the codebase and setting it requires the use of an integer for the corresponding proto generated enum value.
fixes flyteorg/flyte#3664
flyteorg/flytesnacks
GitHub Actions: Mark github pre-release as Release
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
✅ 26 other checks have passed
26/30 successful checks