1 new commit pushed to

<https://github.com/flyteorg/furo/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/furo/commit/8ba17005b9872a42eace45d3f4016eb2c0a9e5b9|8ba17005>

- fix page.html flyteorg/furo

Deployment to github-pages by github-pages[bot] flyteorg/furo

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flyte/commit/88729df96c53d95d944c00fb2684903ea19dba87|88729df9>

- remove getting started (#3298) flyteorg/flyte

#3071 [Docs] Add documentation for Dask integration Issue created by cosmicBboy Description The Dask integration PRs: • flyteorg/flyteidl#339 • flyteorg/flyteplugins#275 • Python PR TBD Should be quickly followed by documentation in flytesnacks showing how to configure the backend plugin and how to use it with flytekit. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#362 Added unmarshal attribute for expires_in for device flow auth token Pull request opened by pmahindrakar-oss Signed-off-by: pmahindrakar-oss prafulla.mahindrakar@gmail.com TL;DR Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue fixes flyteorg/flyte#3288 Follow-up issue flyteorg/flyteidl ✅ All checks have passed 11/11 successful checks

#21 added -y option to docker_build.sh. I also fixed a bug on -p option Pull request opened by masayang I had added a couple of features to the sample template. I also fixed a bug in docker_build.sh • Now -p option works correctly. • By providing -y option, you can specify a version of Python. Default is 3.8 • If pipenv is in the path, requirements.txt will be generated through

pip freeze

• If jupyter is in the path, workflows/*.ipynb will be transformed to *.py files. flyteorg/flytekit-python-template DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#3299 [BUG] Tasks called within tasks should error out when run remotely Issue created by cosmicBboy Describe the bug Currently, tasks within tasks will return a promise, while the same setup on a remote Flyte cluster will invoke the inner task as a plain python function. Consider the following:

@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)

Expected behavior The above code should work when run locally edit: flytekit should also log a warning saying that the inner task is executed as a function and not an atomic entity Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flyte/commit/b4590fa0da0bdb2f5237bc788c2a1ff3c8f99dd9|b4590fa0>

- Update Administrator docs (#3116) flyteorg/flyte

#153 Bump activesupport from 6.0.5.1 to 6.0.6.1 Pull request opened by dependabot[bot] Bumps activesupport from 6.0.5.1 to 6.0.6.1. Release notes Sourced from activesupport's releases.

v6.0.6.1

Active Support

• No changes.

Active Model

• No changes.

Active Record

• Make

sanitize_as_sql_comment

more strict

Though this method was likely never meant to take user input, it was attempting sanitization. That sanitization could be bypassed with carefully crafted input.

This commit makes the sanitization more robust by replacing any occurrances of "/_" or "_/" with "/ " or " /". It also performs a first pass to remove one surrounding comment to avoid compatibility issues for users relying on the existing removal.

This also clarifies in the documentation of annotate that it should not be provided user input.

[CVE-2023-22794]

Action View

• No changes.

Action Pack

• No changes.

Active Job

• No changes.

... (truncated) Commits • `28bb76d` Version 6.0.6.1 • `91cf62e` Version 6.0.6 • `c7d64e9` Preparing for 6.0.5.1 release • `c177e45` updating version and changelog • See full diff in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting

@dependabot rebase

. * * * Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: •

@dependabot rebase

will rebase this PR •

@dependabot recreate

will recreate this PR, overwriting any edits that have been made to it •

@dependabot merge

will merge this PR after your CI passes on it •

@dependabot squash and merge

will squash and merge this PR after your CI passes on it •

@dependabot cancel merge

will cancel a previously requested merge and block automerging •

@dependabot reopen

will reopen this PR if it is closed •

@dependabot close

will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually •

@dependabot ignore this major version

will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this minor version

will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this dependency

will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) •

@dependabot use these labels

will set the current labels as the default for future PRs for this repo and language •

@dependabot use these reviewers

will set the current reviewers as the default for future PRs for this repo and language •

@dependabot use these assignees

will set the current assignees as the default for future PRs for this repo and language •

@dependabot use this milestone

will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the Security Alerts page. flyteorg/flyteorg.github.io ✅ All checks have passed 2/2 successful checks

#1426 Fixes guess type bug in UnionTransformer Pull request opened by kumare3 Signed-off-by: Ketan Umare ketan.umare@gmail.com TL;DR Fixes bug in guesses Union types from remote literal type Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Tracking Issue fixes flyteorg/flyte#3294 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#180 Remove duplication of workflow inputs and outputs #minor Pull request opened by sonjaer TL;DR Currently, users need to define outputs and inputs twice through SdkWorkflow<InputT, OutputT> and in builder.inputOf() and builder.output() which forces users to specify duplicate information which is error prone. This PR removes the ability to specify inputs/outputs through the builder. Tested here: • http://flyteconsole.services.gew1.spotify.net/console/projects/sonjae-test-flyte-scio/domains/development/executions/f64d4bff0192d4ceb84a?duration=all&amp;fromExecutionNav=true • http://flyteconsole.services.gew1.spotify.net/console/projects/sonjae-test-flyte-scio/domains/development/executions/f94605359345d47ca901?duration=all&amp;fromExecutionNav=true • http://flyteconsole.services.gew1.spotify.net/console/projects/sonjae-test-flyte-scio/domains/development/executions/fcb73fe3a3ef94dbd871?duration=all&amp;fromExecutionNav=true Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description See flyteorg/flyte#3303 for more information Tracking Issue Work towards flyteorg/flyte#3303 Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

#314 Only create copilot sidecars if interface requires Pull request opened by hamersaw TL;DR Currently, Flyte injects a copilot init container to download inputs and a sidecar container to upload outputs of a task. This process only checks if the interface has inputs / outputs defined as

nil

, rather then checking if the variable list exists and is empty as well. This means, depending on the definition, Flyte may inject a container to, for example, upload the outputs of a task when that task does not have any outputs. This PR validates variable existence to ensure copilot sidcars are not unnecessarily created. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3254 Follow-up issue NA flyteorg/flyteplugins Codecov: 62.90% (-0.12%) compared to 109224c ✅ 5 other checks have passed 5/6 successful checks

#3300 [Docs] If workflows can save pipeline artifact to s3 instead of tempdir Issue created by esadler-hbo Description Data Scientists developing with cloud notebooks lose temporary files when they shutdown notebooks. Also some companies, like Databricks, create file system abstractions that break the OOS paradigm most software follows. It is possible to use

remote_path

for

StructuredDatasets

, but being able to just run local executions against s3 would mean I would write less code. This might not make sense if the workflow artifacts are hard to interpret out of context. This is a pretty low priority request. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1427 Bump isort to 5.12.0 Pull request opened by eapolinario Signed-off-by: Eduardo Apolinario eapolinario@users.noreply.github.com TL;DR what title says Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description We're seeing issues while installing

isort

in the venv created by pre-commit (e.g. #1426 and #1420). Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#2993 [Housekeeping] Flyte Deployment journey and documentation Issue created by wild-endeavor Revamp Flyte Deployment This ticket encompasses all the work for revamping the Flyte deployment journey. Specifically this covers • Updates to the flyte sandbox-lite image • Changes to flytectl • Documentation updates • Helm chart updates More information to follow. Flyte Local Demo Experience • Persist state to user's local workstation • Run a registry so we can deprecate the --source flag • Preload images New local demo sandbox After this PR is merged, we will have the following structure. Helm • A Helm chart just for the Flyte single binary. • A Helm chart for the sandbox that uses the Flyte single binary Helm chart and pulls in the open source Helm charts for dependencies (like minio/postgres) Images • New bundled image will be created. This is the core of the new experience. It will contain • predownloaded images for all the dependencies including the flyte binary itself. • a registry so that the

--source

flag in flytectl is no longer necessary. Flytectl • Allow users to start with an external flyte config • Allow users to modify a config and reload • Print current state of cluster Docs Implement the new user journey, moving the more difficult error prone things like auth and ingress to later stages. Helm Refactor and introduce new Helm charts that use the single binary. Misc Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/88c6bba6c3b114e9f80fca1768cafd6920935fac|88c6bba6>

- Bump isort to 5.12.0 (#1427) flyteorg/flytekit

#3301 [BUG] version parameter is not being used in `flytectl get task` Issue created by eapolinario Describe the bug

flytectl get task

lists all tasks of a given project and domain. One of the parameters of that command is

--version

(or

--latest

), which allows you to list tasks for a specific version. That parameter is not being honored currently. Expected behavior

flytectl get task --project <project> --domain <domain> --version <version>

should return all tasks for version

<version>

. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#182 [Snyk] Upgrade com.google.errorprone:error_prone_annotations from 2.17.0 to 2.18.0 Pull request opened by EngHabu This PR was automatically created by Snyk using the credentials of a real user. Snyk has created this PR to upgrade com.google.errorprone:error_prone_annotations from 2.17.0 to 2.18.0. ℹ️ Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. * * * • The recommended version is 1 version ahead of your current version. • The recommended version was released 22 days ago, on 2023-01-09. * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs. For more information: https://camo.githubusercontent.com/95e2c20ea69c5bad119ac89ed6dd9b1e14cabbab6e81d0d859fdef59b6fb9efd/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f69497a4f5467324e6a4a6c4e4331684d5745354c54526a4d6a4d74596a5578595331684e4445334e5449334e575a684d446b694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a4d354f4459324d6d55304c57457859546b744e474d794d7931694e5446684c5745304d5463314d6a63315a6d45774f534a3966513d3d 🧐 View latest project report 🛠️ Adjust upgrade PR settings 🔕 Ignore this dependency or unsubscribe from future upgrade PRs flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/ea39054223a9432692c03314b90c45240f2d8419|ea390542>

- Fixes guess type bug in UnionTransformer (#1426) flyteorg/flytekit

#1428 explicitly handle integers for untyped dict type Pull request opened by cosmicBboy fixes #3158 Signed-off-by: Niels Bantilan niels.bantilan@gmail.com TL;DR Protobuf <> json will auto-cast integers to floats when serializing dict values when the type is annotated with a bare

dict

or

typing.Dict

annotation. This will break code that relies on precise types, such as sklearn, which explicitly checks the types of hyperparameter values in Estimator constructor. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Because the interchange between protobuf and json casts ints to floats, this PR introduces a workaround to handle converting floats to integers by parsing the protobug

struct_pb2.Value

with regex and inferring integers based on its

__repr__

, which actually preserves the integer representation, e.g.

number_value: 100

, but accessing the actual

number_value

attribute will still return a float

assert value.number_value == 100.0

. See here in the protobuf codebase where ints and floats are treated the same. Tracking Issue flyteorg/flyte#3158 Follow-up issue NA flyteorg/flytekit GitHub Actions: lint GitHub Actions: docs GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.9, flytekit-whylogs) GitHub Actions: build-plugins (3.9, flytekit-vaex) GitHub Actions: build-plugins (3.9, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.9, flytekit-spark) GitHub Actions: build-plugins (3.9, flytekit-snowflake) GitHub Actions: build-plugins (3.9, flytekit-ray) ✅ 21 other checks have passed 21/30 successful checks

#3295 Handle [re]loading of sandbox configuration via k3s autodeploy Pull request opened by jeevb on <!date^1675062904^{date_short}|2023-01-30T07:15:04Z> flyteorg/flyte ✅ All checks have passed 11/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by sonjaer

<https://github.com/flyteorg/flytekit-java/commit/19f013e1eac252fbba6e3c725f30817562a87974|19f013e1>

- Remove duplication of workflow inputs and outputs #minor (#180) flyteorg/flytekit-java

#3303 [flytekit-java] Remove duplication of workflow inputs and outputs Issue created by narape After #3260 and #3251 were done, currently, flytekit-java users need to define outputs and inputs twice through SdkWorkflow<InputT, OutputT> and in builder.inputOf() and builder.output() which forces users to specify duplicate information which is error prone. This issue is for removing the ability to specify inputs/outputs through the builder. Before

@AutoService(SdkWorkflow.class)
public class WelcomeWorkflow extends SdkWorkflow<WelcomeWorkflow.Input, WelcomeWorkflow.Output> {

  @AutoValue
  public abstract static class Input {
    public abstract SdkBindingData<String> name();
    public static WelcomeWorkflow.Input create(SdkBindingData<String> name) {
      return new AutoValue_WelcomeWorkflow_Input(name);
    }
  }
  
  @AutoValue
  public abstract static class Output {
    public abstract SdkBindingData<String> greeting();
    public static WelcomeWorkflow.Output create(SdkBindingData<String> greeting) {
      return new AutoValue_WelcomeWorkflow_Output(name);
    }
  }
  
  public WelcomeWorkflow() {
    super(
        JacksonSdkType.of(WelcomeWorkflow.Input.class),
        JacksonSdkType.of(WelcomeWorkflow.Output.class));
  }

  @Override
  public void expand(SdkWorkflowBuilder builder) {
    SdkBindingData<String> name = builder.inputOfString("name", "Name to be greeted");

    SdkBindingData<String> greeting =
        builder
            .apply("greet", new GreetTask(), GreetTask.Input.create(name))
            .getOutputs()
            .greeting();

    builder.output("greeting", greeting, "Welcome message");
  }
}

After

@AutoService(SdkWorkflow.class)
public class WelcomeWorkflow extends SdkWorkflow<WelcomeWorkflow.Input, WelcomeWorkflow.Output> {

  @AutoValue
  public abstract static class Input {
    @Description("Name to be greeted")
    public abstract SdkBindingData<String> name();
    public static WelcomeWorkflow.Input create(SdkBindingData<String> name) {
      return new AutoValue_WelcomeWorkflow_Input(name);
    }
  }
  
  @AutoValue
  public abstract static class Output {
    @Description("Welcome message")
    public abstract SdkBindingData<String> greeting();
    public static WelcomeWorkflow.Output create(SdkBindingData<String> greeting) {
      return new AutoValue_WelcomeWorkflow_Output(name);
    }
  }
  
  public WelcomeWorkflow() {
    super(
        JacksonSdkType.of(WelcomeWorkflow.Input.class),
        JacksonSdkType.of(WelcomeWorkflow.Output.class));
  }

  @Override
  public Output expand(SdkWorkflowBuilder builder, Input input) {
    SdkBindingData<String> greeting =
        builder
            .apply("greet", new GreetTask(), GreetTask.Input.create(input.name()))
            .getOutputs()
            .greeting();

   return Output.create(greeting);
  }
}

Also there is the need to add descriptions to autovalue/case classes as the builder methods were the only place to set that up flyteorg/flyte

#183 Add description annotation for scala case classes Pull request opened by narape TL;DR Add description annotation for scala case classes Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description • A new scala annotation was created. • The

SdkScalaType

search for it when reading the case classes fields • When building the variable map it will set up the description or empty String if nothing was found Tracking Issue fixes flyteorg/flyte#3303 Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/542ba84c2926cac074de3d43d724b80dba7821f5|542ba84c>

- Add description annotation for scala case classes (#183) flyteorg/flytekit-java

#149 Adding support opentelemtry to enable performance observability Pull request opened by hamersaw on <!date^1672769809^{date_short}|2023-01-03T18:16:49Z> TL;DR This PR adds wrappers over the opentelementry framework to support exporting distributed traces from Flyte components. The goal is to instrument Flyte microservices to improve performance observability. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytestdlib GitHub Actions: Lint / Run Lint GitHub Actions: Ensure go generate has run ✅ 5 other checks have passed 5/7 successful checks

#3304 [Core feature] Integrate open telemetry into Flyte components Issue created by hamersaw Motivation: Why do you think this is important? OpenTelemetry is a distributed tracing framework designed to ease performance analyses in distributed systems. Inline with our performance observability push, this would provide users a more conclusive understanding of Flyte performance. Additionally, it helps debug performance issues and serves as a benchmarking utility for new features. Goal: What should the final outcome look like, ideally? OpenTelmetry offers many opportunities for instrumentation. We hope to add support for: • grpc connections (ex. FlyteAdmin, datacatalog, FlytePropeller, etc) • blobstore I/O • k8s API server operations • many more Describe alternatives you've considered We have considered two main options: (1) Leaving this as they are: The current state may leave users (or developers) frustrated about system performance with no real explanation. (2) Enhancing prometheus metrics: Flyte currently exposes many metrics through prometheus, however these metics are often aggregations where fine-grained analysis at the workflow / node / or task level is unavailable. Propose: Link/Inline OR Additional context This work is described as "orchestration metrics" in the performance observability RFC. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/471a262a80b5e7cc01727a528ecc6f3f75d12c0b|471a262a>

- Discard fields when replica number equals zero to avoid api client error (#313) flyteorg/flyteplugins

Release - v1.0.32 New release published by flyte-bot Changelog • 471a262 Discard fields when replica number equals zero to avoid api client error (#313) flyteorg/flyteplugins

#359 Inline input data for execution events Pull request opened by katrogan Signed-off-by: Katrina Rogan katroganGH@gmail.com TL;DR Adds parity to the existing flow for inline event output data passing and storage. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description Tested for • Python tasks • Dynamic nodes • Recovered nodes • Branch nodes • Map tasks Tracking Issue flyteorg/flyte#3232 Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks