flyte-org #flyte-github

GitHub

06/16/2023, 11:40 PM

#775 chore: dparent null in alt paths Pull request opened by ursucarina TL;DR Fixes Cannot read property 'scopedId' of undefined Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Fixes Cannot read property 'scopedId' of undefined flyteorg/flyteconsole Codecov: 0.00% of diff hit (target 65.84%) Codecov: 65.80% (-0.04%) compared to 3184646 GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/11 successful checks

GitHub

06/16/2023, 11:55 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/4783c39d7c16ae99f6f9b6d9162cf8ceea1fcbc7|4783c39d>

- chore: dparent null in alt paths (#775) flyteorg/flyteconsole

GitHub

06/17/2023, 3:24 AM

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytesnacks/commit/3d29c2d3601b996075fe4b2df425b305b0e3ba63|3d29c2d3>

- Fix "ray: command not found Error" with Kuberay v0.3.0+ (#1002) flyteorg/flytesnacks

GitHub

06/17/2023, 4:24 AM

Release - flytesnacks v0.3.202 New release published by flyte-bot Changelog • 3d29c2d Fix "ray: command not found Error" with Kuberay v0.3.0+ (#1002) flyteorg/flytesnacks

GitHub

06/17/2023, 8:39 PM

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytesnacks/commit/2a127cf8e3988d0a8a310be5d3c8ec3ca1f2216e|2a127cf8>

- Doc for flyte agent (#971) flyteorg/flytesnacks

GitHub

06/17/2023, 8:39 PM

#971 Doc for flyte agent Pull request opened by pingsutw This doc includes • Overview • how to register a agent in Python • how to build an image for the agent service • How to enable flyte agent https://flyte--971.org.readthedocs.build/projects/cookbook/en/971/auto/core/extend_flyte/agent_service.html flyteorg/flytesnacks GitHub Actions: Serialize & Register Flytesnacks workflow / Register example to sandbox ✅ 29 other checks have passed 29/30 successful checks

GitHub

06/17/2023, 9:33 PM

Release - flytesnacks v0.3.203 New release published by flyte-bot Changelog • 2a127cf Doc for flyte agent (#971) flyteorg/flytesnacks

GitHub

06/19/2023, 5:29 AM

#3789 [BUG] Can't launch workflow when defining `k8sServiceAccount` and `assumableIamRole` Issue created by SuperBo Describe the bug After registering a workflow with following command:

flytectl register files -d development -p flytesnacks --archive flyte-package.tgz --version "0.0.8" --k8sServiceAccount flyte-user --assumableIamRole arn:aws:iam::<acount_name>:role/<role_name>

. I can't launch that workflow, whenever I tried to launch that workflow, they returned this error:

Request failed with status code 500 failed to create workflow in propeller <http://FlyteWorkflow.flyte.lyft.com|FlyteWorkflow.flyte.lyft.com> "atsv5q7rp97m5fdg5kqs" is invalid: [metadata.annotations: Invalid value: "": name part must be non-empty, metadata.annotations: Invalid value: "": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]

Expected behavior Launch Workflow successfully Additional context to reproduce 1. pyflyte init my_project 2. cd my_project 3. pyflyte --pkgs workflows package 4. flytectl register files -d development -p flytesnacks --archive flyte-package.tgz --version "0.0.1" --k8sServiceAccount flyte-user --assumableIamRole arnawsiam:::role/ 5. Execution workflow via Flyte admin or flytectl. Screenshots https://imgur.com/a/DlOYUG6 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

06/19/2023, 9:09 AM

#1697 Feat: Warn when doing local torch elastic training with nnodes > 1 Pull request opened by fg91 TL;DR With

@task(task_config=Elastic(...))

one can perform training with torch elastic launch (

torchrun

). This works both locally as well as in a cluster with a kubeflow

PyTorchJob

. When executing a workflow locally, i.e.

python workflow.py

, but setting e.g.

Elastic(nnodes=2)

, the rendezvous of the workers will timeout because the workers wait for the non-existing workers from the non-existing 2nd node to join. One would have to set the log level to debug in order to see that torch is waiting for the rendezvous to complete. By default, the workflow appears to not do anything. I thins PR I add a warning log message that informs the user about this. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description I check for an environment variable that is set by the kubeflow training operator. If this is not set but the user set

nnodes>1

, the warning is emitted. One could discuss whether we should just automatically switch to

nnodes=1

if the environment variables for distributed training have not been set by the training operator but I found this too intrusive. Warning the user, however, should be done. Tracking Issue NA Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

GitHub

06/19/2023, 4:01 PM

Pending review on flyteorg/flyteplugins - Manage reminder [#347] Fix double http in the Spark Driver UI Link (mucahitkantepe) a month stale · 2 months old · Waiting on _ pingsutw_ [#363] Fix map task cache misses (bstadlbauer) 2 days stale · 3 days old · Waiting on _ hamersaw_

GitHub

06/19/2023, 4:01 PM

Pending review on flyteorg/flyteidl - Manage reminder [#297] Proposal : Settings proto (pmahindrakar-oss) 5 months stale · a year old · Waiting on _ @Haytham Abuelfutuh, katrogan_ [#364] add partition_columns to StructuredDatasetType (cosmicBboy) 4 months stale · 4 months old · Waiting on _ eapolinario_

GitHub

06/19/2023, 4:01 PM

[#577] Add span uri to node event (Yicheng-Lu-llll) 2 days stale · 6 days old · Waiting on _ @Haytham Abuelfutuh, hamersaw, @Ketan (kumare3)_

GitHub

06/19/2023, 4:29 PM

#577 Add a user-specifiable root to upload link request Pull request opened by wild-endeavor TL;DR This is useful for uploading folders. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin ✅ All checks have passed 2/2 successful checks

GitHub

06/19/2023, 5:01 PM

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by fg91

<https://github.com/flyteorg/flytekit/commit/68ac1f5371de1dac96df3f4e9df7ed983cd1c0d5|68ac1f53>

- Warn when training locally with nnodes > 1 (#1697) flyteorg/flytekit

GitHub

06/19/2023, 9:06 PM

#1698 Add requirements to imageSpec Pull request opened by pingsutw TL;DR Add support passing requirements.txt to image spec. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Example

Copy code

from flytekit import task, workflow, ImageSpec

new_flytekit = "git+<https://github.com/flyteorg/flytekit@01f2d63221467face8d35ed0af5308bbb3e66eaa>"
image = ImageSpec(registry="pingsutw", packages=["pandas", new_flytekit], apt_packages=["git"], requirements="requirement.txt")


@task(container_image=image)
def bar():
    print("hello")
    print("hello")


@workflow
def wf() -> str:
    bar()
    return "hi"


if __name__ == '__main__':
    wf()

Tracking Issue NA Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

GitHub

06/20/2023, 10:39 AM

#3790 [Docs] Migrate all flytesnacks examples to use ImageSpec Issue created by samhita-alla Description Currently, when running Flyte tasks and workflows on the Flyte backend, developers need to build a custom Docker image if there are additional dependencies not included in the default flytekit Docker image. To improve the developer experience, this issue aims to add an ImageSpec to all the flytesnacks examples. This will simplify the process and make it more convenient by eliminating the need to build custom Docker images for handling additional dependencies. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

06/20/2023, 10:44 AM

#1963 [Housekeeping] Copy common .github files to `.github` repository Issue created by samhita-alla Describe the issue We have

.github

directories across all repos under Flyte org. If we have a single `.github` repo under flyteorg, we can have all the common

.github

files under one roof. What if we do not do this? Maintenance Related component(s) No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

06/20/2023, 11:22 AM

#3791 [Docs] flytectl demo vs flytectl sandbox Issue created by shekhar-kotekar Description As of now there are 2 ways to create a

fully standalone minimal environment for running Flyte

. One is using

flytectl demo start

command and other is with

flytectl sandbox start

command. Document for demo is here and document for sandbox is here. Having these 2 different ways for the same thing is confusing for a new comer because of multiple reasons: 1. They don't know which is better or preferred way 2. Which is used by Flyte team internally? 3. Is any approach better or worse than the other? 4. Is there any plan to deprecate one of them? If yes then what's the plan / dates, etc? This hard to find document says that

demo

is lighter / better but there isn't any action or ticket to manage this document hence creating this ticket. Hopefully it is not a duplicate issue. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

GitHub

06/20/2023, 12:53 PM

#3792 [Flytectl Feature] flytectl demo cluster - configurable port number Issue created by shekhar-kotekar Problem Description:

flytectl demo start

command fails with below error when a developer has

Docker Desktop

which allows us to have a

Kubernetes

local cluster. Kubernetes API server runs on port

. So we have to either turn off local Kubernetes cluster (not a good option) or allow demo cluster to use some other port.

Copy code

{"status":"Status: Downloaded newer image for <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84%22|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84">}
🧑‍🏭 booting Flyte-sandbox container
😬 Something went wrong: Failed to start Sandbox container 🐋, Please check your docker client and try again. 
Error: Error response from daemon: driver failed programming external connectivity on endpoint flyte-sandbox (5eaf6609c86e6d818291e2f060e42f8002dfd4e288393da6e01109cfa5a60bd7): listen tcp4 0.0.0.0:6443: bind: address already in use

Requested feature: Allow

demo

cluster to use other port preferrably via config file (config.yaml or config-sandbox.yaml or some other file) so that if a developer has some other container / process using the same port, they can change it and quickly get the demo cluster up and running. Environment details: • Docker Desktop with Kubernetes enabled • Mac with M1 chip • Discussion about this issue on slack is here

Copy code

flytectl version  
{
  "App": "flytectl",
  "Build": "0cb2bfa",
  "Version": "0.6.39",
  "BuildTime": "2023-06-20 14:45:22.664715 +0200 CEST m=+0.016518584"
}

flyteorg/flyte

GitHub

06/20/2023, 1:06 PM

#1003 add a note specifying the supported kuberay version Pull request opened by samhita-alla This PR adds a note to the kuberay docs specifying the version supported by the plugin. Fixes flyteorg/flyte#3763 flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

GitHub

06/20/2023, 4:03 PM

#1699 Generate decks at local execution Pull request opened by pingsutw TL;DR this PR breaks it, because it moves

_outpu_deck

entrypoint.py

. we use

entrypoint.py

only when we run the workflow remotely. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue https://unionai.slack.com/archives/C02CMUNT4PQ/p1687268923664029 Follow-up issue NA flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: lint GitHub Actions: Docs Warnings GitHub Actions: docs ✅ 2 other checks have passed 2/25 successful checks

GitHub

06/20/2023, 5:43 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyteidl/commit/283472365c16c0e1e5974a68db98f6620525e90d|28347236>

- Add prefix to data upload (#416) flyteorg/flyteidl

GitHub

06/20/2023, 5:44 PM

Release - v1.5.11 New release published by flyte-bot Changelog • 2834723 Add prefix to data upload (#416) flyteorg/flyteidl

GitHub

06/20/2023, 6:05 PM

#776 [Snyk] Fix for 1 vulnerabilities Pull request opened by EngHabu This PR was automatically created by Snyk using the credentials of a real user. Snyk has created this PR to fix one or more vulnerable packages in the
npm
dependencies of this project. Changes included in this PR • Changes to the following files to upgrade the vulnerable dependencies to a fixed version: • package.json Vulnerabilities that will be fixed With an upgrade: (*) Note that the real score may have changed since the PR was raised. Commit messages Package name: @semantic-release/npm The new version differs by 156 commits. • 410b73d Merge pull request #581 from semantic-release/beta • fc6fd18 Merge branch 'master' of github.com:semantic-release/npm into beta • 93f2f73 chore(deps): lock file maintenance (#590) • 3930bf1 chore(deps): update dependency semantic-release to v20.1.3 (#589) • 4dae1ea build(deps): bump webpack from 5.75.0 to 5.76.1 • 0d914cb chore(deps): update dependency sinon to v15.0.2 (#587) • dff7f90 chore(deps): update dependency dockerode to v3.3.5 (#586) • c6e74d4 chore(deps): lock file maintenance (#585) • c434c47 Merge pull request #582 from semantic-release/provenance • a6aad6f ci(provenance): enabled provenance for publishing • 5fa2115 chore(deps): update dependency semantic-release to v20.1.1 (#584) • 9ad8402 chore(deps): lock file maintenance (#583) • ed0b20f Merge branch 'master' of github.com:semantic-release/npm into beta • 23eb1e3 chore(deps): lock file maintenance (#580) • 3266f40 ci(release): configured the release job to use the latest lts version of node • 725f044 chore(deps): updated semantic-release to the latest stable version • 31bc1f3 Merge pull request #579 from semantic-release/npm-9 • 2a79f80 fix(deps): upgraded npm to v9 • c46c466 chore(deps): lock file maintenance (#573) • f450a28 chore(deps): lock file maintenance (#572) • 2b8ac2e Merge pull request #567 from semantic-release/sr-peer • 1d0ed0f chore(deps): update dependency semantic-release to v20.1.0 (#571) • 22e70ad feat(semantic-release-peer): raised the minimum peer requirement to the first version that supports loading esm plugins • bd05849 chore(deps): update dependency semantic-release to v20.0.4 (#570) See the full diff Package name: jest The new version differs by 250 commits. • be16e47 v27.0.0 • 63102ec chore: update changelog for release • 564694a docs(blog): Jest 27 blog post (#11131) • b68d91b feat(pretty-print): add option

printBasicPrototype

(#11441) • 2226742 chore: minor simplify format results error (#11432) • 78eb25d chore: remove needless assign (#11433) • 696c455 chore: update lockfile after publish • e2eb9ae v27.0.0-next.11 • 3b253f8 Wait for closed resources to actually close before detecting open handles (#11429) • 27bee72 fix: run GC before collecting open handles (#11278) • 50451df feat: use fallback if prettier not found (#11400) • 150dbd8 chore: update lockfile after publish • 6f44529 v27.0.0-next.10 • cbcec7d Upgrade fsevents in jest-haste-map (#11428) • 9633a26 feat: support reporters written in ESM (#11427) • 59f42d8 fix: do not cache modules that throw during evaluation (#11263) • 57e32e9 Detect open hand… flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release GitHub Actions: unit_tests_with_coverage GitHub Actions: lint_project DCO: DCO ✅ 3 other checks have passed 3/9 successful checks

GitHub

06/20/2023, 6:09 PM

#777 fix: launch plan icon Pull request opened by ursucarina TL;DR Fixes Launch plan icon to: • respect selected color • remove additional margins in side nav Before:

image▾

After:

image▾

Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Fixes Launch plan icon to: • respect selected color • remove additional margins in side nav flyteorg/flyteconsole Codecov: 0.00% of diff hit (target 65.80%) GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 7 other checks have passed 7/11 successful checks

GitHub

06/20/2023, 8:26 PM

#1701 test ci Pull request opened by pingsutw TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/7 successful checks

GitHub

06/20/2023, 10:16 PM

#11 Add Yicheng Issue created by pingsutw Highlighted contributions • Flytekit timeline deck • Flytekit runtime metric • Real-time flyte deck • bug fixes for the Ray plugin • https://github.com/flyteorg/flytekit/pulls?q=is%3Apr+author%3AYicheng-Lu-llll What repositories do you expect to keep contributing to? flyteadmin, flytekit, flyteidl, flyteplugins, flytepropeller What is the name of your current organization? UIUC / Influxdata Any other context that you wish to share? cc @Yicheng-Lu-llll flyteorg/community

GitHub

06/21/2023, 5:16 AM

#1702 Add bigquery project and location to metadata Pull request opened by pingsutw TL;DR • Add bigquery project and location to metadata • Return PERMANENT_FAILURE state if there is an error in

BigQurey.job

Fixed https://flyte-org.slack.com/archives/CP2HDHKE1/p1683133259253529 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1683133259253529 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

GitHub

06/21/2023, 6:24 AM

#364 Add docker credential to databricks job config Pull request opened by pingsutw TL;DR Read the docker username and passport from the secret, and add this credential to the Databricks job config Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1686928788919089 Follow-up issue NA flyteorg/flyteplugins GitHub Actions: Run lint ✅ 2 other checks have passed 2/3 successful checks

GitHub

06/21/2023, 3:51 PM

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/4b579560d4d78b0b919a3dc62949850405d9f10d|4b579560>

- fix: launch plan icon (#777) flyteorg/flyteconsole