GitHub
06/21/2023, 4:09 PMGitHub
06/21/2023, 5:39 PMGitHub
06/21/2023, 6:29 PMdocs-simplify
branch so we can test and refine the migration approach.
In summary, this PR moves all the core/flyte_basics
code to a new directory examples/basics
, which is the first step to flattening the example directory structure and making each directory its own stand-alone flyte project.
flyteorg/flytesnacks
GitHub Actions: Publish artifacts to github release
GitHub Actions: Create Prerelease
GitHub Actions: Bump Version
GitHub Actions: Serialize & Register Flytesnacks workflow / Register example to sandbox
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/ray_example)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/pod)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kftensorflow)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfpytorch)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/kfmpi)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_spark)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/kubernetes/k8s_dask)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/gcp/bigquery)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/whylogs_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/sql)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/papermilltasks)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/pandera_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/onnx_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/modin_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/mlflow_example)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/greatexpectations)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/duckdb_examples)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/dolt)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/flytekit_plugins/dbt_example)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/external_services/databricks)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/sagemaker_training)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/sagemaker_pytorch)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/batch)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/integrations/aws/athena)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/examples/basics)
GitHub Actions: Serialize & Register Flytesnacks workflow / Serialize Example (cookbook/core)
✅ No checks have passed
0/30 successful checksGitHub
06/21/2023, 6:46 PMGitHub
06/21/2023, 6:53 PMGitHub
06/21/2023, 7:21 PMGitHub
06/21/2023, 9:48 PM@task
def some_task(inputs: typing.List[int]) -> int:
...
@workflow
def my_wf() -> typing.List[int]
return map_task(some_task, min_success_ratio=0.5)(inputs=[1, 2, 3, 4])
if some of the items fail to process, the resulting output will have shorter length compared to the input list. This is confusing!
Expected behavior
The task either fails to register because the output should have been Optional[int]
or the map task should fail completely because it can't "correctly" produce the output..
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/22/2023, 3:44 AMfrom flytekit import task, workflow, ImageSpec
image_spec = ImageSpec(packages=["numpy"], base_image="private_base_image", registry="private_registry", registry_config="registry_config.json")
@task(container_image=image_spec)
def t1() -> str:
import numpy as np
print("task 1")
print(np.array([1,2,3]))
return("blah1")
@workflow
def wf():
t1()
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
✅ All checks have passed
30/30 successful checksGitHub
06/22/2023, 11:03 AM<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>
by samhita-alla
<https://github.com/flyteorg/flyte-conference-talks/commit/ed0f60a496228b9311c2c0177e9784274afd83db|ed0f60a4>
- updates to code
<https://github.com/flyteorg/flyte-conference-talks/commit/cbcf3e92b198981b88ee72f2cff582f26d061d01|cbcf3e92>
- Merge remote-tracking branch 'origin/main'
flyteorg/flyte-conference-talksGitHub
06/22/2023, 11:30 AM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/0220cda54783a0315345b662edce12ba65495d71|0220cda5>
- Add presentation schedule
flyteorg/communityGitHub
06/22/2023, 12:04 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/9c1a9a842cc7f4451fa8a953d33426292dcc0464|9c1a9a84>
- Add presentation schedule link
flyteorg/communityGitHub
06/22/2023, 7:57 PMmap_value_type
set to an empty LiteralType()
. This makes the compiler think that no bindings are needed. Since dataclasses are inherently multi-variate, we're at a bit of a impasse.
And this PR depends on that. If this approach is untenable, most of this PR should be discarded. (The parts that should be retained are in the Other issues below.) One possible way around this is to make the new dataclass transformer follow the pattern of the existing dataclass json transformer - just expose a scalar, and only interpret the
Background
There's a few places in flytekit core today that blur the line between the type engine and not type engine. These are:
• Extraction of literals from nested promises (translate_inputs_to_literals
) for local execution
• Extraction of literals from nested promises for local workflow execution (ensure_literal
)
• Construction of literals in pyflyte run (one example), there are many others in the file.
This PR attempts to resolve the first two by
• Merging the logic for the first two and moving to the type engine.
• Generalizing what it means to be a container type (see the test here to see more clearly what that implies),
• Remove isinstance(..., list)
and isinstance(..., dict)
• Adding three new functions to the type transformer:
• is_container_type
(returns bool)
• flyte_container_type
(returns LiteralCollection or LiteralMap)
• traverse
(responsible for appropriately walking the structure of the container and returning one of two generators.
• ListGen = typing.Generator[typing.Tuple[typing.Any, typing.Type[T], _type_models.LiteralType], None, None]
• MapGen = typing.Generator[typing.Tuple[str, typing.Any, typing.Type[T], _type_models.LiteralType], None, None]
(Some code has also been left in place to allow flyte_container_type
to return Scalar.)
Other issues
In addition to merging the two very related calls (ensure_literal
and extract_value
), some other bugs/issues uncovered:
• the handling of unions in the extract_value
subroutine was not correct but ensure_literal
was correct.
• we were not correctly walking bindings when the binding was a list or map. That is, if you do
x = t1()
y = t2(a={"key1": x})
the t1
node would not have been listed as an upstream node of n1
flyteorg/flytekit
GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy)
GitHub Actions: build-plugins (3.11, flytekit-polars)
GitHub Actions: build-plugins (3.11, flytekit-papermill)
GitHub Actions: build-plugins (3.11, flytekit-pandera)
GitHub Actions: build-plugins (3.11, flytekit-hive)
GitHub Actions: build-plugins (3.11, flytekit-dolt)
GitHub Actions: build-plugins (3.11, flytekit-deck-standard)
GitHub Actions: build-plugins (3.11, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.11, flytekit-dask)
GitHub Actions: build-plugins (3.11, flytekit-bigquery)
GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.11, flytekit-aws-batch)
GitHub Actions: build-plugins (3.11, flytekit-aws-athena)
GitHub Actions: build-plugins (3.10, flytekit-vaex)
GitHub Actions: build-plugins (3.10, flytekit-snowflake)
GitHub Actions: build-plugins (3.10, flytekit-ray)
GitHub Actions: build-plugins (3.10, flytekit-papermill)
✅ 13 other checks have passed
13/30 successful checksGitHub
06/22/2023, 8:50 PMGitHub
06/22/2023, 8:59 PMGitHub
06/22/2023, 9:23 PMn1
here
def my_wf(a: int, b: str) -> (int, str):
x, y = t1(a=a)
d = t2(a={"key1": b, "key2": y})
return x, d
would not have picked up n0
as an upstream node because we weren't correctly traversing the structure.
Found while implementing the now rejected generalized container types https://github.com/flyteorg/flytekit/pull/1700/files
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Instead of iterating through the kwargs again, save nodes found while traversing the bindings from the promises.
Tracking Issue
NA
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: docs
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: lint
GitHub Actions: Docs Warnings
✅ 2 other checks have passed
2/17 successful checksGitHub
06/22/2023, 9:27 PMGitHub
06/22/2023, 11:00 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyteadmin/commit/9b22c0e12b858ff6e6accb5e415b83cb3df62ee8|9b22c0e1>
- Add a user-specifiable root to upload link request (#577)
flyteorg/flyteadminGitHub
06/22/2023, 11:07 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/007e63605be094b3efee8fb0bddee5a266e8b16a|007e6360>
- WIP: Add localStorage for selected Project/Domain (#774)
flyteorg/flyteconsoleGitHub
06/22/2023, 11:33 PMGitHub
06/22/2023, 11:42 PMGitHub
06/23/2023, 4:54 AMGitHub
06/23/2023, 6:16 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytesnacks/commit/d641c106b404cb3663199dede7a153c0eaa4cca8|d641c106>
- Increase memory of horovod_train_task (#1005)
flyteorg/flytesnacksGitHub
06/23/2023, 8:01 AMGitHub
06/23/2023, 9:08 AMdf[df.total_vaccinations.notnull()]
to df
, so the example can work.
image▾
GitHub
06/23/2023, 9:13 AMimage▾
GitHub
06/23/2023, 9:34 AMGitHub
06/23/2023, 10:27 AMGitHub
06/23/2023, 1:52 PMGitHub
06/23/2023, 4:58 PM--image name=<image_uri>
raises and error in a pyflyte serialize
invocation:
Failed with Unknown Exception <class 'docker_image.reference.ReferenceInvalidFormat'> Reason: invalid reference format
pyflyte package
can handle this format.
Expected behavior
pyflyte serialize
should have the same behavior as pyflyte package
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/23/2023, 6:22 PM