#3498 Streaming File and Directory in flytekit Issue created by cosmicBboy flyteorg/flyte

#3499 [BUG] Flyteconsole: mapped tasks in execution detail panel tabs not updating to correct node Issue created by jsonporter Describe the bug When viewing a mapped task's execution detail panel, clicking on one of the mapped tasks should update the context of the panel to that specific execution. However, the details within the tabs (Inputs, Outputs, Tasks) still display the details from the parent execution. Expected behavior When a user is viewing a mapped task in the execution detail panel and clicks into one of the sub/mapped tasks, the context for all of the details in the panel should be set to that mapped tasks execution. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/0a681cd1a9e2c6586e6408532767539be65737f4|0a681cd1>

- feat: add support for inter-cluster communication for Ray plugin (#321) flyteorg/flyteplugins

Release - v1.0.42 New release published by flyte-bot Changelog • 0a681cd feat: add support for inter-cluster communication for Ray plugin (#321) flyteorg/flyteplugins

#3500 [BUG] Status of Sub-Workflow not being updated correctly. Issue created by stephen37 Describe the bug When a workflow is running with sub-workflows, even if the workflow is in a

Failed

state, the sub-workflows can stay in a

Queued

state in the UI (cf. screenshot) Those are impossible to stop as they are technically not running anymore. Here is the list of pods in the namespace associated with this project.

❯ k get pods 
NAME                        READY   STATUS      RESTARTS   AGE
ajrs5vwphfsx2sj9ldmq-n0-0   0/1     Completed   0          175m
ajrs5vwphfsx2sj9ldmq-n1-0   0/1     Completed   0          175m
ajrs5vwphfsx2sj9ldmq-n2-0   0/1     Completed   0          175m
ajrs5vwphfsx2sj9ldmq-n3-0   0/1     Completed   0          173m
aq8pt9rjwpssjw7whvsq-n0-0   0/1     Completed   0          24h
aq8pt9rjwpssjw7whvsq-n1-0   0/1     Completed   0          24h
avp8s7qxhgtpw7ckz2gv-n0-0   0/1     Completed   0          10h
avp8s7qxhgtpw7ckz2gv-n1-0   0/1     Completed   0          10h
azc4p4n9g9k8q4bdg767-n0-0   0/1     Completed   0          10h
azc4p4n9g9k8q4bdg767-n1-0   0/1     Completed   0          10h

Discussion in Slack with @hamersaw https://flyte-org.slack.com/archives/CP2HDHKE1/p1679075564278029 Expected behavior The UI should also be showing the sub-workflows in

Failed

stage or

Aborted

stage I guess depending on what the real states of the sub-worklows are. Additional context to reproduce No response Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3501 Overlay execution info messages in timeline view Issue created by hamersaw Discussed in #3429 Originally posted by hamersaw March 8, 2023 Motivation The timeline UI view is marginally useful to debug performance, but has a lot of room for improvement. Integrating the runtime metrics breakdown proposed in the performance observability RFC is a step in the right direction, partitioning node executions into a collection of categorized time-series. This representation will help the "what" but misses a lot of the "why". For example, if a particular execution has a large amount of frontend plugin overhead this means that Flyte started the Task but the backend service has not yet indicated the service has started. K8s gurus will be quick to identify that there may be scheduling contention, large image pull times, or a few other likely scenarios. However, this is not easily available to the user even though FlytePropeller has this information available. We currently store a singular "reason" for the current execution status' but may be better off tracking a time-series of reasons to better explain the execution. Proposal This proposal outlines a solution for overlaying a collection of human readable messages in the timeline view. The exact representation is VERY open for debate, but I envision something similar to jaeger (time-series telemetry data with events) which uses a single tick mark that displays a message on hover. This solution supplies the "why" in an explanation of the reported execution status that will complement the "what" in the runtime breakdown of the execution time-series. The goal will be to balance utility with simplicity, displaying a "useful" number of messages to improve context. Implementation Currently, FlyteAdmin maintains a singular "reason" within the task execution metadata. This is updated in-place on each event from FlytePropeller, meaning the old "reasons" are not persisted. At risk of over-simplifying this, we will need to transition to maintaining a collection of "reasons" with associated timestamps. This will require updates in the following repositories: • FlyteIDL: update TaskExecutionClosure to have repeated reasons with associated timestamps. • FlyteAdmin: use an append to the "reason" list rather than overwriting the existing singular "reason". • FlyteConsole: correctly parse the "reason" list to annotate the timeline UI view. Open Questions 1. How should this be visualized? I will leave this discussion for more UI / UX oriented personnel. 2. Should we add this information to node executions / workflow executions? Currently the "reason" is only tracked for the task-level execution. 3. Do we need to be able to send multiple reasons in a single task event? currently possible to skip phases if execution progresses before FlytePropeller detects and processes the intermediate stage could use event buffers to just send multiple events -> probably the better solution flyteorg/flyte

#1556 [WIP] Partial support in flytekit - especially map tasks Pull request opened by kumare3 TL;DR Partial's support Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#3403 flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-scikitlearn) GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: lint GitHub Actions: Docs Warnings WIP: Title contains "WIP" ✅ 1 other check has passed 1/29 successful checks

1 new commit pushed to

<https://github.com/flyteorg/datacatalog/tree/master|master>

by hamersaw

<https://github.com/flyteorg/datacatalog/commit/25c0a10b18ba0cdd72867168a5e5eca8f22d9b96|25c0a10b>

- replaced github.com/satori/go.uuid with github.com/gofrs/uuid (#101) flyteorg/datacatalog

Release - v1.0.44 New release published by flyte-bot Changelog • 25c0a10 replaced github.com/satori/go.uuid with github.com/gofrs/uuid (#101) flyteorg/datacatalog

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/93995e29dcd37317e91ebcfb3019a4ffed0e82de|93995e29>

- Update the pypi wait (#1554) flyteorg/flytekit

#3503 [Docs] Update K8s plugins that use the kubeflow operators Issue created by cosmicBboy Description This docs page needs to be updated so that users install https://github.com/kubeflow/training-operator instead of the now-unmaintained repos for the separate operators (e.g. https://github.com/kubeflow/pytorch-operator) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#940 Workflow input ordering and description Issue created by ajsalow Motivation: Why do you think this is important? The current UI for workflow input would be more user friendly if workflow inputs had additional descriptions other than their name. Additionally, inputs are sorted alphabetically but user might want arrange inputs in a specific order. Goal: What should the final outcome look like, ideally? Don't sort inputs, keep in order that user defines in workflow. Parse doc string of workflow for input description and include in UI if found. Describe alternatives you've considered [Optional] Propose: Link/Inline OR Additional context If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document. flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/98e74c273a61fd767ea331f5f0812d22a4f7aa57|98e74c27>

- Stream Directories and Files using Flyte (#1512) flyteorg/flytekit

#1512 Stream Directories and Files using Flyte Pull request opened by kumare3 TL;DR This PR implements streaming FlyteFile and allows walking FlyteDirectory. the streaming FlyteFile returns a Filelike object that is supported by fsspec. Note this is still experimental and we would love feedback on the API. Example: Accept a flytefile and stream to another FlyteFile, using simple caching

@task
def copy_file(ff: FlyteFile) -> FlyteFile:
    new_file = FlyteFile.new_remote_file(ff.remote_path)
    with ff.open("r", cache_type="simplecache", cache_options={}) as r:
        with new_file.open("w") as w:
            w.write(r.read())
    return new_file

Example: Accept a flyteDirectory and walk it, while copying the directory file by file to another Flyte directory

@task
def process_folder(fd: FlyteDirectory) -> FlyteDirectory:
    out_fd = FlyteDirectory.new_remote()
    for base, x in fd.crawl():
        src = os.path.join(base, x)
        out_file = out_fd.new_file(x)
        with FlyteFile(src).open("rb") as f:
            with out_file.open("wb") as o:
                o.write(f.read())
    return out_fd

Alternatively, Files can always be handled by fsspec streaming

@task
            def copy_file(ff: FlyteFile) -> FlyteFile:
                new_file = FlyteFile.new_remote_file(ff.name)
                fs = flytekit.get_filesystem()
                with fs.open(f"readahead::{ff.remote_path}", "rb", readahead={}) as r:
                    with new_file.open("wb") as w:
                        w.write(r.read())
                return new_file

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description Other minor changes: • Get rid of sqlite data download dependency by adding it to this repo • Make the base to_html function not abstract. Tracking Issue flyteorg/flyte#3334 flyteorg/flytekit Codecov: 41.48% of diff hit (target 69.70%) Codecov: 69.31% (-0.39%) compared to 93995e2 ✅ 28 other checks have passed 28/30 successful checks

Release - [Beta] FlyteFile and FlyteDirectory streaming support New release published by wild-endeavor What's Changed • prefer FLYTE_ prefixed AWS creds env vars by @flixr in #1523 • Filter out remote entity when generating pb by @honnix in #1545 • Pyflyte docs formatting by @bryanwweber in #1538 • Data subsystem by @wild-endeavor in #1526 • Less strict docker dependency versions by @jcugat in #1536 • Pluck retry from flytekit and into sagemaker by @eapolinario in #1411 • Update the pypi wait by @wild-endeavor in #1554 • Stream Directories and Files using Flyte by @kumare3 in #1512 New Contributors • @bryanwweber made their first contribution in #1538 • @jcugat made their first contribution in #1536 Full Changelog: v1.4.2...v1.4.3b0 flyteorg/flytekit

#727 fix: show correct i/o in details panel Pull request opened by ursucarina TL;DR Map task i/o tabs in details panel don't show correct info when a task is selected Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Map task i/o tabs in details panel don't show correct info when a task is selected flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/9 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteadmin/commit/dea5b2a8b21d1f6edd7fc4b4c5288cbef102075d|dea5b2a8>

- bumping fosite version (#538) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteidl/commit/0f18851e644b1ee51fcf6d4fac855a65adcc34f8|0f18851e>

- Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382) flyteorg/flyteidl

Release - v1.3.13 New release published by flyte-bot Changelog • 0f18851 Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382) flyteorg/flyteidl

#540 Tracking reasons time-series Pull request opened by hamersaw DO NOT MERGE waiting on PRs: flyteorg/flyteidl#382 TL;DR This PR populates the new

Reasons

field on the

TaskExecutionClosure

to track a reason time-series. This will be very useful in improving visualizations by tracking the reason for task phase transitions and updates. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3501 Follow-up issue NA flyteorg/flyteadmin GitHub Actions: Unit Tests / Run Unit Test GitHub Actions: Check Go Generate / Go Generate GitHub Actions: Lint / Run Lint GitHub Actions: Docker Build Images / Build Docker Image ✅ 2 other checks have passed 2/6 successful checks

#520 Abort tasks on non-success phase Pull request opened by hamersaw TL;DR FlytePropeller fast-fails when aborting tasks in a terminal phase. The result is that

Pods

marked as a retryable failure are never cleaned up. This PR only fast-fails when aborting tasks in a success phase, to ensure that all resources are correctly cleaned up at the risk of unnecessary API calls (ex. object does not exist, etc). Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Currently, the

TaskHandler

fast-fails when aborting resources in a terminal phase. This causes issues where resources fail to cleanup when the task is reported as failed (ie. retryable failure or permanent failure) but the resource remains running. One such scenario is handling

Pods

in the "ImagePullBackoff" state. We mark these as failed, but because of the fast-fail during abort these k8s resources are never deleted. This may occur within other plugins as well. To be safe, we may want to attempt to abort all task phases other than

Success

. Tracking Issue fixes flyteorg/flyte#3239 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

Release - v1.1.84 New release published by flyte-bot Changelog • dea5b2a bumping fosite version (#538) flyteorg/flyteadmin

#1553 Pass locally defined scopes to RemoteClientConfigStore Pull request opened by franco-bocci TL;DR This PR fixes a bug in the auth mechanism where locally defined

scopes

are ignored and the ones defined in the admin are always used. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description If

scopes

are locally defined, use those for auth. This mirrors the reference implementation here. Tracking Issue flyteorg/flyte#3486 Follow-up issue flyteorg/flytekit Codecov: 69.60% (-0.10%) compared to 34f80ba ✅ 29 other checks have passed 29/30 successful checks

#3052 [Housekeeping] Remove unused retry dependency in flytekit Issue created by hajapy Describe the issue Remove the unused

retry

library dependency declared by flytekit: https://github.com/flyteorg/flytekit/blob/3b5d15acb5389d99b86ba342a77e1cab5ce8546d/setup.py#L68 This library is not used anywhere within flytekit that I can find searching for

import retry

or

from retry

, except in the sagemaker plugin: https://github.com/flyteorg/flytekit/blob/3b5d15acb5389d99b86ba342a77e1cab5ce8546d/plugins/flytekit-aws-sagemaker/flytekitplugins/awssagemaker/distributed_training.py#L8 The dependency could be moved to the sagemaker plugin's requirements so only users of that plugin need it installed. Though, given the unmaintained state of the retry library, I'd advise against that and suggest to look for a maintained alternative (reretry? https://github.com/leshchenko1979/reretry) What if we do not do this? All users of flytekit end up installing an unmaintained library (see invl/retry#36, not updated since 2016), which also ends up bringing with it unnecessary dependency on

py

which itself has been the source of lots of security alerting pytest-dev/py#287 Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3505 [Core feature] improve pyflyte usage error for missing projects Issue created by cosmicBboy Motivation: Why do you think this is important? When I do

pyflyte run --remote -p nonexistent_project ...

the error message that's produced contains a whole lot of information about the task while not reporting what is actually wrong, which is that

nonexistent_project

doesn't exist in the remote Flyte cluster. See here for full context on what the current error message is. Goal: What should the final outcome look like, ideally? Consider specifying an invalid project or domain as simple input errors and should fail quickly with explicit messaging i.e. "Project 'flytesnacks' not found". A full stack trace is hard to mentally parse for such a simple input error even if it did actually say why it failed in the message. Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3506 [Core feature] Improve error message when there's a type conversion error Issue created by cosmicBboy Motivation: Why do you think this is important? When the flyte compiler returns a type mismatch error, the exact task and workflows names don't appear. For example, consider the workflow:

from flytekit import task, workflow


@task
def t1(x: int) -> int:
    return x + 1


@task
def t2(x: str) -> str:
    return str(x)


@workflow
def wf(x: int) -> str:
    out1 = t1(x=x)
    return t2(x=out1)


if __name__ == "__main__":
    print(wf(x=5))

The error:

File "/Users/nielsbantilan/git/flytekit/flytekit/core/type_engine.py", line 185, in to_python_value
    raise TypeTransformerFailedError(f"Cannot convert literal {lv} to {self._type}")
flytekit.core.type_engine.TypeTransformerFailedError: Cannot convert literal scalar {
  primitive {
    integer: 6
  }
}
 to <class 'str'>

Goal: What should the final outcome look like, ideally? When there's a type mismatch error, identify where exactly in the workflow it's happening: • the task where the conversion failed • the input that fails conversion Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#728 Install deps directly in checks.yml Pull request opened by eapolinario TL;DR Install dependencies using yarn directly Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description As suggested by bahmutov/npm-install#146 (comment), we can leverage

actions/setup-node@v3

to cache dependencies. Tracking Issue N/A Follow-up issue NA flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 8 other checks have passed 8/11 successful checks

#974 fix formatting on waiting_for_external_inputs.py Pull request opened by cosmicBboy fixes a minor formatting issue that caused the html rendering to be incorrect: flyteorg/flytesnacks GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dolt) GitHub Actions: Build & Push to GHCR (cookbook/integrations/flytekit_plugins/dbt_example) GitHub Actions: Build & Push to GHCR (cookbook/integrations/external_services/databricks) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_training) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/sagemaker_pytorch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/batch) GitHub Actions: Build & Push to GHCR (cookbook/integrations/aws/athena) GitHub Actions: Build & Push to GHCR (cookbook/core) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/spark_horovod) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/pima_diabetes) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/nlp_processing) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/mnist_classifier) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/ml_training/house_price_prediction) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/feast_integration) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/feature_engineering/eda) GitHub Actions: Build & Push to GHCR (cookbook/case_studies/bioinformatics/blast) GitHub Actions: Docs Warnings ✅ 3 other checks have passed 3/20 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteconsole/commit/d0d383e6e0036a9596222e37dd6e0e014dd8e99b|d0d383e6>

- Install deps directly in checks.yml (#728) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyte/commit/d30e0b5943411db51bb62df4a1ee26f75e9fc12f|d30e0b59>

- [RFC] Performance Benchmarking (#2995) flyteorg/flyte