1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/8ce2da311a9a0b84dc403dc960e286cd74d6fec1|8ce2da31>

- Update material (#6) flyteorg/flyte-conference-talks

#7 Bump ipython from 8.7.0 to 8.10.0 in /scipy-2023 Pull request opened by dependabot[bot] Bumps ipython from 8.7.0 to 8.10.0. Commits • `15ea1ed` release 8.10.0 • `560ad10` DOC: Update what's new for 8.10 (#13939) • `7557ade` DOC: Update what's new for 8.10 • `385d693` Merge pull request from GHSA-29gw-9793-fvw7 • `e548ee2` Swallow potential exceptions from showtraceback() (#13934) • `0694b08` MAINT: mock slowest test. (#13885) • `8655912` MAINT: mock slowest test. • `a011765` Isolate the attack tests with setUp and tearDown methods • `c7a9470` Add some regression tests for this change • `fd34cf5` Swallow potential exceptions from showtraceback() • Additional commits viewable in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting

@dependabot rebase

. * * * Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: •

@dependabot rebase

will rebase this PR •

@dependabot recreate

will recreate this PR, overwriting any edits that have been made to it •

@dependabot merge

will merge this PR after your CI passes on it •

@dependabot squash and merge

will squash and merge this PR after your CI passes on it •

@dependabot cancel merge

will cancel a previously requested merge and block automerging •

@dependabot reopen

will reopen this PR if it is closed •

@dependabot close

will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually •

@dependabot ignore this major version

will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this minor version

will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this dependency

will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page. flyteorg/flyte-conference-talks GitHub Actions: build (scipy-2023, 3.10, macos-latest) GitHub Actions: build (scipy-2023, 3.10, ubuntu-latest) GitHub Actions: build (scipy-2023, 3.9, windows-latest) GitHub Actions: build (scipy-2023, 3.9, macos-latest) GitHub Actions: build (scipy-2023, 3.9, ubuntu-latest) GitHub Actions: build (scipy-2023, 3.8, windows-latest) GitHub Actions: build (scipy-2023, 3.8, macos-latest) GitHub Actions: build (scipy-2023, 3.8, ubuntu-latest) ✅ 2 other checks have passed 2/10 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/34e13309beda07686c3dfd5e8841f32e9808943f|34e13309>

- update readme flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/758b748a34475c0d3701c81d0affdfe2ea75a6d2|758b748a>

- Add July 11 meeting notes flyteorg/community

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/d16d36e9536ab34cc525bd44817a20bd9affc81d|d16d36e9>

- add mlflow, clean up notebook flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/d029720888de0b4f292de97a14e05fb88e6561ef|d0297208>

- remove ipdb flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/2407e5c5c13108e3253aa69583c3e4a3c4c375f9|2407e5c5>

- update launchplan flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/4eb8155f0713b7a24d64140e0c696ea3f70b4da5|4eb8155f>

- notebook updates flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/6f12394ad208c95695181605efd9ce41cf5ab9cc|6f12394a>

- Update contributor ladder (#15) flyteorg/community

#413 Expose buildkit port Pull request opened by jeevb flyteorg/flytectl GitHub Actions: Test Getting started GitHub Actions: Dry Run Goreleaser ✅ 2 other checks have passed 2/4 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyte/commit/951a93080972bbcf85bc92ec54ac62e074be887f|951a9308>

- Update Flyte components (#3819) flyteorg/flyte

Release - v1.8.0 New release published by eapolinario What's Changed • Support configuring pip index url with the image spec by @yini7777 in #1692 • Improve error message for pyflyte-fast-execute by @pingsutw in #1686 • Feat: Allow disabling rich tracebacks via env var by @fg91 in #1695 • Fix task type version in pytorch and mpi task plugin by @yubofredwang in #1690 • get_transformer returns pickle transformer if type is unsupported by @pingsutw in #1694 • Add Cuda to ImageSpec by @pingsutw in #1688 • Feat: Warn when doing local torch elastic training with nnodes > 1 by @fg91 in #1697 • Fix config of user facing execution parameters in spawning elastic tasks by @fg91 in #1677 • Union | optional return types supported by @kumare3 in #1703 • Fail registration if output isn't Optional when using map tasks with min_success_ratio < 1 by @pingsutw in #1709 • Add bigquery project and location to metadata by @pingsutw in #1702 • Generate decks at local execution by @pingsutw in #1699 • Improve task type hint by @pingsutw in #1711 • Detect upstream nodes from container nested promises by @wild-endeavor in #1707 • Add Iterator Transformer by @pingsutw in #1710 • Always a default image and streamline image handling serialize vs package by @wild-endeavor in #1610 • Remove ensure literal by @wild-endeavor in #1706 • Fix a bug in gx integration by @XinEDprob in #1675 • Update pythonbuild.yml: add hugging face plugin to CI by @cosmicBboy in #1684 • add imagespec cache by @RichhLi in #1717 • Add requirements to imageSpec by @pingsutw in #1698 • Memory reduction change by @wild-endeavor in #1716 • use getattr to access task_def.disable_deck in entrypoint by @cosmicBboy in #1724 • Add a couple tests by @wild-endeavor in #1722 • Csvtransform by @ChungYujoyce in #1671 • add private registry support by @RichhLi in #1705 • update docs around pyflyte run by @samhita-alla in #1728 • fix Image Spec requirements.txt hash by @RichhLi in #1731 New Contributors • @yini7777 made their first contribution in #1692 • @XinEDprob made their first contribution in #1675 • @RichhLi made their first contribution in #1717 Full Changelog: v1.7.0...v1.8.0 flyteorg/flytekit

Release - Flyte v1.8.0 milestone release New release published by flyte-bot Flyte v1.8.0 Release Flytekit • Support configuring pip index url with the image spec by @yini7777 in flyteorg/flytekit#1692 • Improve error message for pyflyte-fast-execute by @pingsutw in flyteorg/flytekit#1686 • Feat: Allow disabling rich tracebacks via env var by @fg91 in flyteorg/flytekit#1695 • Fix task type version in pytorch and mpi task plugin by @yubofredwang in flyteorg/flytekit#1690 • get_transformer returns pickle transformer if type is unsupported by @pingsutw in flyteorg/flytekit#1694 • Add Cuda to ImageSpec by @pingsutw in flyteorg/flytekit#1688 • Feat: Warn when doing local torch elastic training with nnodes > 1 by @fg91 in flyteorg/flytekit#1697 • Fix config of user facing execution parameters in spawning elastic tasks by @fg91 in flyteorg/flytekit#1677 • Union | optional return types supported by @kumare3 in flyteorg/flytekit#1703 • Fail registration if output isn't Optional when using map tasks with min_success_ratio < 1 by @pingsutw in flyteorg/flytekit#1709 • Add bigquery project and location to metadata by @pingsutw in flyteorg/flytekit#1702 • Generate decks at local execution by @pingsutw in flyteorg/flytekit#1699 • Improve task type hint by @pingsutw in flyteorg/flytekit#1711 • Detect upstream nodes from container nested promises by @wild-endeavor in flyteorg/flytekit#1707 • Add Iterator Transformer by @pingsutw in flyteorg/flytekit#1710 • Always a default image and streamline image handling serialize vs package by @wild-endeavor in flyteorg/flytekit#1610 • Remove ensure literal by @wild-endeavor in flyteorg/flytekit#1706 • Fix a bug in gx integration by @XinEDprob in flyteorg/flytekit#1675 • Update pythonbuild.yml: add hugging face plugin to CI by @cosmicBboy in flyteorg/flytekit#1684 • add imagespec cache by @RichhLi in flyteorg/flytekit#1717 • Add requirements to imageSpec by @pingsutw in flyteorg/flytekit#1698 • Memory reduction change by @wild-endeavor in flyteorg/flytekit#1716 • use getattr to access task_def.disable_deck in entrypoint by @cosmicBboy in flyteorg/flytekit#1724 • Add a couple tests by @wild-endeavor in flyteorg/flytekit#1722 • Csvtransform by @ChungYujoyce in flyteorg/flytekit#1671 Flyteadmin • Add a user-specifiable root to upload link request by @wild-endeavor in flyteorg/flyteadmin#577 • Bump propeller version by @pingsutw in flyteorg/flyteadmin#580 • Upgrade go 1.19 in Dockerfile by @hamersaw in flyteorg/flyteadmin#581 Flytepropeller • Update flyteplugins to v1.0.67 by @bstadlbauer in flyteorg/flytepropeller#575 • Correctly validating error code on aborting terminal launchplans by @hamersaw in flyteorg/flytepropeller#574 • Use GetExecutionData instead by @EngHabu in flyteorg/flytepropeller#573 • Propagating environment variables through launchplans by @hamersaw in flyteorg/flytepropeller#576 • upgrade dockerfile go to 1.19 by @hamersaw in flyteorg/flytepropeller#580 • Bump flyteplugins version by @pingsutw in flyteorg/flytepropeller#581 • Support for cross-project secrets for GCP by @jeevb in flyteorg/flytepropeller#582 • Added IsFailurePermanent flag on DynamicTaskStatus by @hamersaw in flyteorg/flytepropeller#567 Flyteplugins • Don't add master replica log link when doing elastic pytorch training by @fg91 in flyteorg/flyteplugins#356 • [Bigquery] Add support for impersonation of GSA bound to task's KSA by @jeevb in flyteorg/flyteplugins#355 • Fix initial dask job state by @bstadlbauer in flyteorg/flyteplugins#357 • Feat: Add pod start and finish time in RFC3339 time format to logging link templating variables #minor by @fg91 in flyteorg/flyteplugins#360 • fix v1 pytorch job plugin with elastic policy by @yubofredwang in flyteorg/flyteplugins#359 • Allow using pod start time in kubeflow plugin log links by @fg91 in flyteorg/flyteplugins#362 • Pass location to the BigQuery request by @pingsutw in flyteorg/flyteplugins#365 • Fix map task cache misses by @bstadlbauer in flyteorg/flyteplugins#363 • Switch to official dask operator by @bstadlbauer in flyteorg/flyteplugins#366 • Fix duplicate env vars in container by @hamersaw in flyteorg/flyteplugins#358 • Set scheduler restart policy to Always by @bstadlbauer in <https://github.com/flyteorg/flyteplugins/pull/367|flyteorg/flyteplugins#367… flyteorg/flyte

Deployment to github-pages by flyte-bot flyteorg/flyte

#784 feat: url breadcrumbs feature Pull request opened by FrankFlitton TL;DR Change the page header to be consistent across the app with interactive breadcrumbs. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description A row of breadcrumbs will appear above the page header. Each one may contain a drop down with links to similar items, eg workflows linking to other workflow pages. Project and domain switching is also supported. The last item in the list will be treated as a page header. A breadcrumb list can be provided in the config for custom breadcrumbs. The approach is to parse the URL for even odd key value pairs. Then, the url segment, as well as complete URL and the next and previous segments, are passed to a validator within each bread crumb object that controls if it should render or not for a given URL. The value the breadcrumb displays is either directly from the URL or evaluated at runtime. A breadcrumb object must be supplied or no breadcrumb will be rendered for a given URL segment. Todo: link to design docs. Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA flyteorg/flyteconsole DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#1733 Update the comments so that use ImageConfig.auto can support all the use case. Pull request opened by Future-Outlier TL;DR Update the comments so that use ImageConfig.auto can support all the use case. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description I am using the flyte documentation, and I found that flytesnacks doesn't teach how to specify ImageConfig. After tracing the code in ImageConfig, I found that the old use case (default str) is not a good option. And I am going to update the flytesnacks ! flyteorg/flytekit DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#1734 Update ImageConfig outdated usage in comment Pull request opened by Future-Outlier TL;DR Update the comments so that use ImageConfig.auto can support all the use case. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description I am using the flyte documentation, and I found that flytesnacks doesn't teach how to specify ImageConfig. After tracing the code in ImageConfig, I found that the old use case (default str) is not a good option. And I am going to update the flytesnacks ! flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-huggingface) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/25 successful checks

#1024 Fix image config in getting started Pull request opened by Future-Outlier flyteorg/flytesnacks ✅ All checks have passed 2/2 successful checks

#371 Check if ResourceMeta exists before sending the request Pull request opened by pingsutw Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1689004671152499 Follow-up issue NA flyteorg/flyteplugins ✅ All checks have passed 2/2 successful checks

#3862 [BUG] Flytekit Contribution dependency doesn't work Issue created by Future-Outlier Describe the bug Hi, community I just follow the contribution guide to install the dependency to contribute. However, I encounter 2 environment erros and doesn't know how to solve it. Can anyone guide me ? I would love to contribute. I use Python 3.9.17 as my environment and simple just run.

conda create env -n flytekit  python=3.9
make setup
pip install -e .
pip install gsutil awscli

and run

make requirements

The below will appear the logs the first one:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorboard 2.8.0 requires google-auth-oauthlib<0.5,>=0.4.1, but you have google-auth-oauthlib 1.0.0 which is incompatible.
Successfully installed setuptools-68.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <https://pip.pypa.io/warnings/venv>
pip-compile --upgrade --verbose <http://doc-requirements.in|doc-requirements.in>
/root/miniconda3/envs/build/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Using pip-tools configuration defaults found in 'pyproject.toml'.
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.

the second one

Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches grpcio!=1.55.0,<2.0,<2.0dev,<=1.49.1,<=1.51.3,>=1.24.3,>=1.32.0,>=1.33.2,>=1.47.0,>=1.50.0 (from -r <http://doc-requirements.in|doc-requirements.in> (line 4))
Tried: 0.13.0, 0.13.1, 0.14.0, 0.15.0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.1.3, 1.2.0, 1.2.1, 1.3.0, 1.3.5, 1.4.0, 1.6.0, 1.6.3, 1.7.0, 1.7.3, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.6, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.11.0, 1.11.1, 1.12.0, 1.12.1, 1.13.0, 1.14.0, 1.14.1, 1.14.2, 1.15.0, 1.16.0, 1.16.1, 1.17.0, 1.17.1, 1.18.0, 1.19.0, 1.20.0, 1.20.1, 1.21.1, 1.22.0, 1.22.1, 1.23.0, 1.23.1, 1.24.0, 1.24.1, 1.24.3, 1.25.0, 1.26.0, 1.27.1, 1.27.2, 1.28.1, 1.29.0, 1.30.0, 1.31.0, 1.32.0, 1.33.1, 1.33.1, 1.33.1, 1.33.2, 1.33.2, 1.33.2, 1.34.0, 1.34.0, 1.34.0, 1.34.1, 1.34.1, 1.34.1, 1.35.0, 1.35.0, 1.35.0, 1.36.0, 1.36.0, 1.36.0, 1.36.1, 1.36.1, 1.36.1, 1.37.0, 1.37.0, 1.37.0, 1.37.1, 1.37.1, 1.37.1, 1.38.0, 1.38.0, 1.38.0, 1.38.1, 1.38.1, 1.38.1, 1.39.0, 1.39.0, 1.39.0, 1.40.0, 1.40.0, 1.40.0, 1.41.0, 1.41.0, 1.41.0, 1.41.1, 1.41.1, 1.41.1, 1.42.0, 1.42.0, 1.42.0, 1.43.0, 1.43.0, 1.43.0, 1.44.0, 1.44.0, 1.44.0, 1.45.0, 1.45.0, 1.45.0, 1.46.0, 1.46.0, 1.46.0, 1.46.1, 1.46.1, 1.46.1, 1.46.3, 1.46.3, 1.46.3, 1.46.5, 1.46.5, 1.46.5, 1.47.0, 1.47.0, 1.47.2, 1.47.2, 1.47.5, 1.47.5, 1.48.0, 1.48.0, 1.48.1, 1.48.1, 1.48.2, 1.48.2, 1.49.0, 1.49.0, 1.49.1, 1.49.1, 1.50.0, 1.50.0, 1.51.0, 1.51.0, 1.51.1, 1.51.1, 1.51.3, 1.51.3, 1.52.0, 1.52.0, 1.53.0, 1.53.0, 1.53.1, 1.53.1, 1.54.0, 1.54.0, 1.54.2, 1.54.2, 1.55.0, 1.55.0, 1.56.0, 1.56.0
Skipped pre-versions: 0.4.0a0, 0.4.0a1, 0.4.0a2, 0.4.0a3, 0.4.0a4, 0.4.0a5, 0.4.0a6, 0.4.0a7, 0.4.0a8, 0.4.0a13, 0.4.0a14, 0.5.0a0, 0.5.0a1, 0.5.0a2, 0.9.0a0, 0.9.0a1, 0.10.0a0, 0.11.0b0, 0.11.0b1, 0.12.0b0, 0.13.1rc1, 0.14.0rc1, 1.0.0rc1, 1.0.0rc2, 1.0.1rc1, 1.9.0rc1, 1.9.0rc2, 1.9.0rc3, 1.10.0rc2, 1.10.1rc1, 1.10.1rc2, 1.11.0rc1, 1.11.0rc2, 1.11.1rc1, 1.12.0rc1, 1.13.0rc1, 1.13.0rc2, 1.13.0rc3, 1.14.0rc1, 1.14.0rc2, 1.14.2rc1, 1.15.0rc1, 1.16.0rc1, 1.20.0rc1, 1.20.0rc2, 1.20.0rc3, 1.21.0rc1, 1.21.1rc1, 1.22.0rc1, 1.23.0rc1, 1.24.0rc1, 1.25.0rc1, 1.26.0rc1, 1.27.0rc1, 1.27.0rc2, 1.28.0rc1, 1.28.0rc2, 1.34.0rc1, 1.34.0rc1, 1.34.0rc1, 1.35.0rc1, 1.35.0rc1, 1.35.0rc1, 1.36.0rc1, 1.36.0rc1, 1.36.0rc1, 1.37.0rc1, 1.37.0rc1, 1.37.0rc1, 1.38.0rc1, 1.38.0rc1, 1.38.0rc1, 1.39.0rc1, 1.39.0rc1, 1.39.0rc1, 1.40.0rc1, 1.40.0rc1, 1.40.0rc1, 1.41.0rc2, 1.41.0rc2, 1.41.0rc2, 1.42.0rc1, 1.42.0rc1, 1.42.0rc1, 1.43.0rc1, 1.43.0rc1, 1.43.0rc1, 1.44.0rc1, 1.44.0rc1, 1.44.0rc1, 1.44.0rc2, 1.44.0rc2, 1.44.0rc2, 1.45.0rc1, 1.45.0rc1, 1.45.0rc1, 1.46.0rc1, 1.46.0rc1, 1.46.0rc1, 1.46.0rc2, 1.46.0rc2, 1.46.0rc2, 1.47.0rc1, 1.47.0rc1, 1.48.0rc1, 1.48.0rc1, 1.49.0rc1, 1.49.0rc1, 1.49.0rc3, 1.49.0rc3, 1.50.0rc1, 1.50.0rc1, 1.51.0rc1, 1.51.0rc1, 1.52.0rc1, 1.52.0rc1, 1.53.0rc2, 1.53.0rc2, 1.54.0rc1, 1.54.0rc1, 1.55.0rc1, 1.55.0rc1, 1.56.0rc2, 1.56.0rc2
There are incompatible versions in the resolved dependencies:
  grpcio<=1.49.1 (from -r <http://doc-requirements.in|doc-requirements.in> (line 4))
  grpcio<2.0dev,>=1.47.0 (from google-cloud-bigquery==3.11.3->-r <http://doc-requirements.in|doc-requirements.in> (line 27))
  grpcio<2.0,>=1.24.3 (from tensorflow==2.8.1->-r <http://doc-requirements.in|doc-requirements.in> (line 44))
  grpcio<=1.51.3,>=1.32.0 (from ray==2.5.1->-r <http://doc-requirements.in|doc-requirements.in> (line 47))
  grpcio<2.0dev,>=1.33.2 (from google-api-core[grpc]==2.11.1->-r <http://doc-requirements.in|doc-requirements.in> (line 17))
  grpcio!=1.55.0,<2.0,>=1.50.0 (from flytekit==0.0.0+develop->-r <http://doc-requirements.in|doc-requirements.in> (line 1))
make: *** [Makefile:65: doc-requirements.txt] Error 2

Expected behavior Build successfully. Additional context to reproduce No response Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3863 [BUG] FlyteConsole can't handle the example merge_sort.py Issue created by Future-Outlier Describe the bug When I use the example from merge_sort.py, which is from here Use the following command

pyflyte run --remote merge_sort.py merge_sort --numbers=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21] --numbers_count=21

and open my flyteconsole, the bug will happen. The output is right, but the behavior isn't. Note: I've contributed to the merge_sort.py to change the behavior, but my code can still reproduce the bug. Expected behavior 1.See the graph in FlyteConsole. 2.See the Nodes in FlyteConsole. Additional context to reproduce No response Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3864 [Core feature] Enable dataclass attribute access in workflows Issue created by franzigeiger Motivation: Why do you think this is important? I'd like to access dataclass attributes from within a workflow. This is already possible for

NamedTuple

but those are restricted in usage. Goal: What should the final outcome look like, ideally? I would like this to work:

@dataclass
@dataclass_json
class Config:
    x:int
    y:int
    z:int
    some_unused_additional_config:list[str]

@task
def add(x:int, y:int):
    return x+y
@workflow
def do_something(config:Config):
    add(config.x, config.y)

Describe alternatives you've considered Currenlty this is only possible with workarounds. You can: • Make the workflow dynamic • Use

NamedTuples

only if the configuration object is an output of another workflow. They can't be a task or workflow input. • Change the

add

task signatures to take in the full config object, which means taking in unused parameters. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3865 [BUG] Dynamic task name override does not work when multiple tasks are created Issue created by gpgn Describe the bug Task names can be overridden with

.with_overrides(node_name="foo")

. This is useful for dynamic tasks especially, because they get cryptic identifiers such as

d1-n-dn0

by default, making it difficult to see in the UI for what iteration a dynamically created task is running. If multiple sub-tasks are created, the name override works initially for all tasks, but then fails for any tasks after the initial one. Expected behavior Overridden task names are visible throughout in the UI. Additional context to reproduce

from flytekit import Resources, WorkflowFailurePolicy, dynamic, task, workflow


@task(
    limits=Resources(mem="500Mi"),
)
def some_task(
    val: str,
):
    print(val)


@dynamic(limits=Resources(mem="500Mi"))
def base_workflow():
    # set up task promises
    for val in ["one", "two", "three"]:
        task_some_task = some_task(
            val=val,
        ).with_overrides(node_name=f"{val}-some-task")
        task_some_other_task = some_task(
            val=val,
        ).with_overrides(node_name=f"{val}-some-other-task")

        task_some_task >> task_some_other_task


@workflow(    failure_policy=WorkflowFailurePolicy.FAIL_AFTER_EXECUTABLE_NODES_COMPLETE,
)
def wf():
    base_workflow()

Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#424 Fix: Allow `nproc_per_node` in `task_config=Elastic(...` to be string Pull request opened by fg91 on <!date^1689166837^{date_short}|2023-07-12T13:00:37Z> Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

#1736 Fix: Remove broken option to pass str value as `nproc_per_node` in `task_config=Elastic` Pull request opened by fg91 TL;DR

torchrun

has an argument called

--nproc-per-node

which can be either

"auto"

,

"cpu"

,

"gpu"

or an integer (see here). The argument controls the size of the local worker groups on each worker node. Accordingly, the

ElasticPolicy

of the

PyTorchJob

CRD in the kubeflow training operator (which is used by the

flytekitplugins.kfpytorch

plugin) states:

type ElasticPolicy struct {
        ...
	// Number of workers per node; supported values: [auto, cpu, gpu, int].
        ...

When building the torch elastic/torchrun Flyte plugin, I translated this into the following `@task(task_config=...)`:

@dataclass
class Elastic(object):
    """
    Configuration for `torch elastic training <https://pytorch.org/docs/stable/elastic/run.html>`_.
    Use this to run single- or multi-node distributed pytorch elastic training on k8s.
    ...
    Args:
        nproc_per_node (Union[int, str]): Number of workers per node. Supported values are [auto, cpu, gpu, int].
    """"

* * * However, the kubeflow training operator makes a mistake here which I unfortunately didn't notice and propagated into flytekit:

type ElasticPolicy struct {
        ...
	// Number of workers per node; supported values: [auto, cpu, gpu, int].
	NProcPerNode *int32 `json:"nProcPerNode,omitempty"`
        ...

The type does not allow string values despite the comment directly above. If a user of the Flyte elastic plugin passes a string in their task decorator, registration currently fails since the respective message in

flyteidl

also expects an int32. * * * • I created an issue to fix this in the kubeflow training operator: kubeflow/training-operator#1861 • In this PR, I remove the option to pass a string value to

@task_config=Elastic(nproc_per_node=...)

because this didn't work in the first place unfortunately. In case this will be fixed in the training operator, I will fix this here. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by fg91

<https://github.com/flyteorg/flytekit/commit/2602cb55e038523c43e5b672d26e0f1358b9b3d6|2602cb55>

- Remove option to pass str value as nproc_per_node in elastic config (#1736) flyteorg/flytekit

#3866 [Core feature] Parallelize Node Executions in FlytePropeller Issue created by hamersaw Motivation: Why do you think this is important? Currently, FlytePropeller uses a modified DFS to traverse the FlyteWorkflow DAG where each node evaluation is synchronous. This means that in certain scenarios DAG traversal can be a bottleneck. For example, extremely large workflows or tasks that operate synchronously. Goal: What should the final outcome look like, ideally? FlytePropeller should evaluate nodes in parallel. This could be done adding a layer of futures on top of the current traversal algorithm so that node scheduling remains consistent but the actual evaluation is done in the background. Describe alternatives you've considered Leaving it as is may result in poor performance in certain scenarios as described. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3665 [BUG] Flytekit 1.5.0 uses far more RAM than flytekit 1.4.2 Issue created by ossareh Describe the bug I've documented this in slack: https://flyte-org.slack.com/archives/CREL4QVAQ/p1683729924011689 Duplicated here: For the same set of inputs with the only difference between: • flytekit version • schema change (container tag + k8s resource requests) Flytekit 1.5.0 uses FAR more RAM. Based on the changelog my best guess is that s3fs is using memory where the older mechanism in 1.4.2 was using disk. This seems like a bad default when considering the cost basis of each resource. Expected behavior Minimal to no difference in memory pressure for the same workloads when switching between flytekit versions. Additional context to reproduce Fetch same data (in our case ~12Gb) using flytekit 1.4.2 and flytekit 1.5.0 and observe system memory usage. Screenshots Flytekit 1.4.2 Flytekit 1.5.0 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/8f41e67ddfeb8cde2aef231a5ce9a23ee7b0ee98|8f41e67d>

- Alter ID sequence to bigint (#578) flyteorg/flyteadmin

Release - v1.2.15 New release published by eapolinario As mentioned in the 1.2.14 release notes, official support for protobuf3 ended in 23Q2. This release backports the 1.6.1 and 1.6.2 releases, concluding then our support for protobuf3. Only critical or security fixes will warrant another 1.2.x release going forward. What's Changed • Backport 1.6.1 and 1.6.2 to 1.2 by @eapolinario in #1729 Full Changelog: v1.2.14...v1.2.15 flyteorg/flytekit