GitHub
05/12/2023, 8:07 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytekit/commit/71d78988752cf804e3fb05f9d1f6321ca8b2146b|71d78988>
- Improve workflow decorator type hints with overload (#1635)
flyteorg/flytekitGitHub
05/12/2023, 8:56 PMGitHub
05/12/2023, 9:14 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/88f6d15cd2c0b434600dfcb6ff631d7803abb0c2|88f6d15c>
- Update Flyte components (#3675)
flyteorg/flyteGitHub
05/12/2023, 9:26 PMGitHub
05/12/2023, 10:13 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/d391691c6db314da7298520e4fc83b2f5fe01eb9|d391691c>
- Update Flyte components (#3677)
flyteorg/flyteGitHub
05/12/2023, 10:33 PMparent_id
column into bigint
only if necessary by @eapolinario in flyteorg/flyteadmin#554
Propeller
• Moved controller-runtime start out of webhook Run function by @hamersaw in flyteorg/flytepropeller#546
• Fixing recovering of SKIPPED nodes by @hamersaw in flyteorg/flytepropeller#551
• Remove resource injection on the node for container task by @ByronHsu in flyteorg/flytepropeller#544
• Infer GOOS and GOARCH from environment by @jeevb in flyteorg/flytepropeller#552
• fix makefile to read variables from environment and overrides by @jeevb in flyteorg/flytepropeller#554
• Remove BarrierTick by @hamersaw in flyteorg/flytepropeller#545
• Check for TerminateExecution error and eat Precondition status by @EngHabu in flyteorg/flytepropeller#553
• Setting primaryContainerName by default on Pod plugin by @hamersaw in flyteorg/flytepropeller#555
• Implement ability to specify additional/override annotations when using Vault Secret Manager by @pradithya in flyteorg/flytepropeller#556
• Maintaining Interruptible and OverwriteCache for reference launchplans by @hamersaw in flyteorg/flytepropeller#557
• Added support for aborting task nodes reported as failures by @hamersaw in flyteorg/flytepropeller#541
• Added support for EnvironmentVariables on ExecutionConfig by @hamersaw in flyteorg/flytepropeller#558
• Fast fail if task resource requests exceed k8s resource limits by @hamersaw in flyteorg/flytepropeller#488
New Contributors
• @wckdman made their first contribution in flyteorg/flytekit#1588
• @peridotml made their first contribution in flyteorg/flytekit#1599
• @ringohoffman made their first contribution in flyteorg/flytekit#1631
• @elibixby made their first contribution in flyteorg/flytekit#1615
• @ByronHsu made their first contribution in flyteorg/flytepropeller#544
flyteorg/flyteGitHub
05/12/2023, 10:35 PMGitHub
05/12/2023, 11:08 PMnode_mapping
. we can directly get task interface from the node_mapping
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☑︎ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
^^^
Follow-up issue
NA
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-modin)
GitHub Actions: build-plugins (3.8, flytekit-mlflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.8, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.8, flytekit-hive)
GitHub Actions: build-plugins (3.8, flytekit-greatexpectations)
GitHub Actions: build-plugins (3.8, flytekit-envd)
GitHub Actions: build-plugins (3.8, flytekit-duckdb)
GitHub Actions: build-plugins (3.8, flytekit-dolt)
GitHub Actions: build-plugins (3.8, flytekit-deck-standard)
GitHub Actions: build-plugins (3.8, flytekit-dbt)
GitHub Actions: build-plugins (3.8, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: lint
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: Docs Warnings
GitHub Actions: docs
✅ 1 other check has passed
1/30 successful checksGitHub
05/12/2023, 11:13 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/dab1eed3b3dc29e2587c2473ede218ab97c904a4|dab1eed3>
- Multi arch imageSpec (#1630)
flyteorg/flytekitGitHub
05/14/2023, 1:17 AMGitHub
05/14/2023, 2:31 AMGitHub
05/14/2023, 2:59 AMpy.typed
marker as per PEP 0561
Expected behavior
Usage of flytekit is checked against its types.
Additional context to reproduce
Given
from typing import Dict, List, NamedTuple
from flytekit import task
class OpenFlightsData(NamedTuple):
routes: List[Dict[str, str]]
airlines: Dict[str, str]
airports: Dict[str, Dict[str, str]]
@task()
def extract_reference_data() -> OpenFlightsData:
return OpenFlightsData(
routes=[{"MAO": "CIZ"}], airlines={"0B": "Blue Air"}, airports={"AAA": {"lat": "1", "lon": "2"}}
)
Then when running mypy 1.3.0:
❯ mypy aircraft/test.py
aircraft/test.py:3: error: Skipping analyzing "flytekit": module is installed, but missing library stubs or py.typed marker [import]
aircraft/test.py:3: note: See <https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports>
Found 1 error in 1 file (checked 1 source file)
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/14/2023, 3:54 AMtask
return type was changed to effectively Tuple[Promise] | Promise | VoidPromise | Tuple[Unknown, ...] | None
.
This breaks usage of tasks in type checkers like pyright / vscode.
Expected behavior
Task function calls still type check correctly, as they did in flytekit 1.5.0
Additional context to reproduce
This type checks without errors in flytekit 1.5.0:
from typing import Dict, List, NamedTuple
import pandas as pd
from flytekit import task, workflow
class OpenFlightsData(NamedTuple):
routes: List[Dict[str, str]]
airlines: Dict[str, str]
airports: Dict[str, Dict[str, str]]
@task()
def extract_reference_data() -> OpenFlightsData:
return OpenFlightsData(
routes=[{"MAO": "CIZ"}], airlines={"0B": "Blue Air"}, airports={"AAA": {"lat": "1", "lon": "2"}}
)
@task()
def extract_live_data() -> pd.DataFrame:
return pd.DataFrame()
@task()
def transform(raw_aircraft_data: pd.DataFrame, airlines: Dict[str, str]) -> pd.DataFrame:
# do stuff ..
return pd.DataFrame()
@workflow
def main() -> None:
reference_data = extract_reference_data()
reveal_type(reference_data) # expecting OpenFlightsData
live_data = extract_live_data()
reveal_type(live_data) # expecting Dataframe
transform(raw_aircraft_data=live_data, airlines=reference_data.airlines)
if __name__ == "__main__":
main()
With flytekit 1.6.0 and pyright / vscode:
❯ pyright test.py
test.py
test.py:34:17 - information: Type of "reference_data" is "Tuple[Promise] | Promise | VoidPromise | Tuple[Unknown, ...] | None"
test.py:36:17 - information: Type of "live_data" is "Tuple[Promise] | Promise | VoidPromise | Tuple[Unknown, ...] | None"
test.py:38:68 - error: Cannot access member "airlines" for type "Tuple[Promise]"
Member "airlines" is unknown (reportGeneralTypeIssues)
test.py:38:68 - error: Cannot access member "airlines" for type "Promise"
Member "airlines" is unknown (reportGeneralTypeIssues)
test.py:38:68 - error: Cannot access member "airlines" for type "VoidPromise"
Member "airlines" is unknown (reportGeneralTypeIssues)
test.py:38:68 - error: Cannot access member "airlines" for type "Tuple[Unknown, ...]"
Member "airlines" is unknown (reportGeneralTypeIssues)
test.py:38:68 - error: "airlines" is not a known member of "None" (reportOptionalMemberAccess)
5 errors, 0 warnings, 2 informations
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
05/14/2023, 4:09 AMpy.typed
file so that when type checkers are run in codebases that use flytekit, its types will be checked.
Resolves flyteorg/flyte#3681
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Add py.typed
file
Tracking Issue
flyteorg/flyte#3681
flyteorg/flytekit
GitHub Actions: build-plugins (3.8, flytekit-pandera)
GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn)
GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-modin)
GitHub Actions: build-plugins (3.8, flytekit-mlflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow)
GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch)
GitHub Actions: build-plugins (3.8, flytekit-kf-mpi)
GitHub Actions: build-plugins (3.8, flytekit-k8s-pod)
GitHub Actions: build-plugins (3.8, flytekit-hive)
GitHub Actions: build-plugins (3.8, flytekit-greatexpectations)
GitHub Actions: build-plugins (3.8, flytekit-envd)
GitHub Actions: build-plugins (3.8, flytekit-duckdb)
GitHub Actions: build-plugins (3.8, flytekit-dolt)
GitHub Actions: build-plugins (3.8, flytekit-deck-standard)
GitHub Actions: build-plugins (3.8, flytekit-dbt)
GitHub Actions: build-plugins (3.8, flytekit-data-fsspec)
GitHub Actions: build-plugins (3.8, flytekit-dask)
GitHub Actions: build-plugins (3.8, flytekit-bigquery)
GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker)
GitHub Actions: build-plugins (3.8, flytekit-aws-batch)
GitHub Actions: build-plugins (3.8, flytekit-aws-athena)
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: Docs Warnings
✅ No checks have passed
0/30 successful checksGitHub
05/14/2023, 6:16 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/8ef79e535521500a4de6b84de268b323a580129b|8ef79e53>
- Add executor_path and applications_path to spark config (#1634)
flyteorg/flytekitGitHub
05/14/2023, 2:36 PMGitHub
05/15/2023, 4:38 AMaudience
parameter into the get_token
method - via the ClientCredentials flow and the ClientConfig object. Also adds in the ability for Pyflyte/Flytekit to read in the audience from the config.yaml file, but it is not linked to the get_token method (only ClientConfig object params are used at the moment).
Type
• [ x ] Bug Fix
• [ x ] Feature
☐ Plugin
Are all requirements met?
• [ x ] Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
• [ x ] Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Auth0 requires an audience parameter for its various requests. This was previously not enabled in the Flytekit source code. This PR adds support for that by piping in the audience
parameter as input from FlyteAdmin's audience field, and then into the various functions/methods/classes that handle the auth request.
The ability to pipe in "audience" from the config.yaml file is also enabled so set up is there for enabling this method of defining the parameter, but it's fine if it gets deleted as the ClientConfig inferring audience from Admin works.
Tracking Issue
flyteorg/flyte#3661
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
DCO: DCO
✅ 1 other check has passed
1/2 successful checksGitHub
05/15/2023, 5:02 AMaudience
parameter into the get_token
method - via the ClientCredentials flow and the ClientConfig object. Also adds in the ability for Pyflyte/Flytekit to read in the audience from the config.yaml file, but it is not linked to the get_token method (only ClientConfig object params are used at the moment).
Type
• [ x ] Bug Fix
• [ x ] Feature
☐ Plugina
Are all requirements met?
• [ x ] Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
• [ x ] Any pending items have an associated Issue
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Auth0 requires an audience parameter for its various requests. This was previously not enabled in the Flytekit source code. This PR adds support for that by piping in the audience
parameter as input from FlyteAdmin's audience field, and then into the various functions/methods/classes that handle the auth request.
The ability to pipe in "audience" from the config.yaml file is also enabled so set up is there for enabling this method of defining the parameter, but it's fine if it gets deleted as the ClientConfig inferring audience from Admin works.
Tracking Issue
flyteorg/flyte#3661
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
05/15/2023, 12:19 PMGitHub
05/15/2023, 12:49 PMGitHub
05/15/2023, 2:35 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/4b1f0a2c5e1c8ede233c0c8622f243b44122e7a8|4b1f0a2c>
- feat: Allow "audience" param to be set in values.yaml - Flyte-Binary (000-core.yaml) (#3683)
flyteorg/flyteGitHub
05/15/2023, 2:52 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/b410108887b76b5afcd7cfd36aaeda7540737513|b4101088>
- Add support for env vars to pyflyte run (#1617)
flyteorg/flytekitGitHub
05/15/2023, 3:00 PMmap_task(a_mappable_task)
and map_task(a_mappable_task, concurrency=2)
may both be used at the same time
☐ retries hard-coded to 1 on the node-level so task-level not used
☐ support for partials
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Extremely WIP at the moment.
Need to verify support for container tasks and other backend entities.
Tracking Issue
flyteorg/flyte#1131
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
WIP: Title contains "WIP"
✅ 1 other check has passed
1/2 successful checksGitHub
05/15/2023, 3:02 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/22ed97fc892d1fa2575746106dfc2f9bdcd68ba2|22ed97fc>
- Export Flytedecks support for TLRO (#757)
flyteorg/flyteconsoleGitHub
05/15/2023, 3:26 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytesnacks/commit/b3f67cdc22a2528ccf10c1eb6f8ad09caa65fd3e|b3f67cdc>
- Update vault secret manager documentation and add example (#990)
flyteorg/flytesnacksGitHub
05/15/2023, 4:00 PMGitHub
05/15/2023, 4:00 PMGitHub
05/15/2023, 4:36 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyteadmin/commit/610451d21ca2fa8f598ebc26b8ebb96e423e270a|610451d2>
- Inject user identifier to ExecutionSpec (#549)
flyteorg/flyteadminGitHub
05/15/2023, 4:36 PMGitHub
05/15/2023, 4:46 PM<https://github.com/flyteorg/flyte/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyte/commit/6fd4ddb5354314038a85ddb32375da1aa077bd3b|6fd4ddb5>
- feat: allow seedProjects to be set via inline configuration (#3631)
flyteorg/flyte