#3246 [Core feature] DuckDB Integration Issue created by samhita-alla Motivation: Why do you think this is important? DuckDB integration can be helpful to run queries on files, dataframes, pyarrow tables in an efficient manner. A native integration with Flyte opens up a new set of possibilities to use DuckDB from within an orchestration platform, which is the first of its kind. Goal: What should the final outcome look like, ideally? A task plugin that enables users to run queries seamlessly; along the lines of the following prototype:

duckdb_task = DuckDBQuery(name="duckdb_task", query="SELECT SUM(a) FROM mydf", inputs=kwtypes(mydf=pd.DataFrame))

Describe alternatives you've considered An alternative is to handle DuckDB code from within a Flyte task: https://gist.github.com/samhita-alla/003c3f409e8caa88470f6f7206b54ae3. Propose: Link/Inline OR Additional context A task plugin that accepts a query, a dataframe/pyarrow table/parquet file/csv file and parameters. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1419 DuckDB plugin Pull request opened by samhita-alla Signed-off-by: Samhita Alla aallasamhita@gmail.com TL;DR This PR adds a

DuckDBQuery

task plugin that runs queries using DuckDB as the DBMS. Type ☐ Bug Fix ☐ Feature ☑︎ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description Capturing the crucial assumptions I made while building the task plugin: • The

DuckDBQuery

task parameter that a user needs to send argument to includes

query

and can contemplate adding includes

inputs

. •

query

can include a set of queries that'll be run sequentially. The last query needs to be a SELECT query. •

inputs

can include structured dataset or a list of parameters to be sent to the queries. • The

output

is a pyarrow table. Can be converted to any structured dataset compatible type. • The connection mode is set to

:memory

, i.e., the data is always stored in an in-memory, non-persistent database. It can be set to a file, but it's difficult to make the file accessible to different

DuckDBQuery

pods, which otherwise wouldn't make sense because file is persistent, and it needs to be leveraged. Example:

duckdb_query = DuckDBQuery(
    name="read_parquet",
    query=[
        "INSTALL httpfs",
        "LOAD httpfs",
        """SELECT hour(lpep_pickup_datetime) AS hour, count(*) AS count FROM READ_PARQUET(?) GROUP BY hour""",
    ],
    inputs=kwtypes(params=list[str]),
)

@workflow
def parquet_wf(parquet_file: str) -> pd.DataFrame:
    return duckdb_query(params=[parquet_file])

assert isinstance(
    parquet_wf(parquet_file="<https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-02.parquet>"),
    pd.DataFrame,
)

Tracking Issue Fixes flyteorg/flyte#3246 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3284 [Housekeeping] Improve the way requirements are regenerated Issue created by cosmicBboy Describe the issue TODO regenerating requirements: • 1 requirements files per python version? • tox? • dockerize? • how to manage plugins requirements? What if we do not do this? TODO Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3373 [Docs] Document how to run Flyte deployment via Unionoss-maintained terraform files Issue created by cosmicBboy Description See: #3283 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3374 Link to dataflow execution Issue created by jonathanschuchart Motivation: Why do you think this is important? It would make the flyte console even more complete as a central go-to for a job execution if I were able to go from the flyte execution graph view to the dataflow graph. Goal: What should the final outcome look like, ideally? Clicking on the dataflow job node in the graph should provide the link to the dataflow page Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3375 Linking to flyte job execution page Issue created by jonathanschuchart Motivation: Why do you think this is important? If I want to get to the execution page of a flyte job (like this one, I have to go through the "Project Dashboard" page, instead of by clicking through the workflow links in the "Workflows" page. The last level in the Workflows page doesn't link to anything anymore, even though it shows an (unclickable) execution graph. While going through the Project Dashboard page is arguably faster, it is confusing and unexpected. Goal: What should the final outcome look like, ideally? When starting at "Workflows" page and clicking through to the list of executions, the last level should link to the actual execution page with all the details. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

a day stale · 2 days old · Waiting on _ eapolinario, @Niels Bantilan, @Yee, @Ketan (kumare3), pingsutw_

#3376 update swagger link in flyte demo Pull request opened by cosmicBboy Describe your changes Update the link to swapper openapi docs due to changes in Flyte sandbox Check all the applicable boxes ☑︎ I updated the documentation accordingly. • [ ] All new and existing tests passed. ☑︎ All commits are signed-off. flyteorg/flyte ✅ All checks have passed 6/6 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flyte/commit/7f66e475b522745f3721170906db32f61d9af357|7f66e475>

- update swagger link in flyte demo (#3376) flyteorg/flyte

#1524 [WIP] Backend plugin system (grpc) Pull request opened by pingsutw TL;DR WIP Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#3282 Follow-up issue NA flyteorg/flytekit GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings GitHub Actions: docs WIP: Title contains "WIP" ✅ 1 other check has passed 1/6 successful checks

#198 Literaltype as sdktype Pull request opened by narape TL;DR Use SdkLiteralType to create SdkType with low overhead. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description Currently the way to create custom

SdkType<T>

in Java is using

JacksonSdkType

. For example If a developer wants to define a task equivalent to (String str) -> str.toUpperCase() currently he/she needs to write it as

@AutoService(SdkRunnableTask.class)
public class ToUpperCaseTask extends SdkRunnableTask<ToUpperCaseTask.Input, ToUpperCaseTask.Output> {
  public ToUpperCaseTask() {
    super(JacksonSdkType.of(Input.class), JacksonSdkType.of(Output.class));
  }

  @AutoValue
  public abstract static class Input {
    public abstract SdkBindingData<String> in();
  }

  @AutoValue
  public abstract static class Output {
    public abstract SdkBindingData<String> out();

    public static Output create(String out) {
      return new AutoValue_ToUpperCaseTask_Output(out);
    }
  }

  @Override
  public Output run(Input input) {
    return Output.create(SdkBindingDataFactory.of(input.get().toUpperCase()));
  }
}

For such trivial task, defining the AutoValues to define the types takes the majority of the lines. With this PR the developer could write the task this way instead

@AutoService(SdkRunnableTask.class)
public class ToUpperCaseTask extends SdkRunnableTask<SdkBindingData<String>, SdkBindingData<String>> {
  public ToUpperCaseTask() {
    super(SdkTypes.of(SdkLiteralTypes.strings(), "str"), SdkTypes.of(SdkLiteralTypes.strings(), "upr"));
  }

  @Override
  public SdkBindingData<String> run(SdkBindingData<String> input) {
    return input.get().toUpperCase();
  }
}

Defining the types for the task now takes one line only and the run is simplified too as no need to create or access AutoValues Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit-java DCO: DCO ✅ 1 other check has passed 1/2 successful checks

#530 Empty structs for task resource defaults Pull request opened by wild-endeavor TL;DR Currently if you don't specify anything, these defaults will spin up pods with

limits:
        cpu: "2"
        memory: 200Mi
      requests:
        cpu: "2"
        memory: 200Mi

Removing this will make it

resources:
      limits:
        cpu: "0"
        memory: "0"
      requests:
        cpu: "0"
        memory: "0"

which means that there is no limit Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Bump Version ✅ 8 other checks have passed 8/12 successful checks

#1823 [Core][flytekit] TypeTransformer support for Pydantic Models Issue created by kumare3 Motivation: Why do you think this is important? Currently Flytekit supports dataclasses as supported user defined types. We should also support Pydantic, as it provides similar benefits as Dataclasses_json to extract the schema and serialize to JSON with a nice extensible format. Goal: What should the final outcome look like, ideally?

from typing import List
from pydantic import BaseModel


class Foo(BaseModel):
    count: int
    size: float = None

Support as a valid transform Describe alternatives you've considered dataclasses are already supported, but they allow limited extensibility to schema extraction as they use marshmallow underneath Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#155 Extends SdkTypes to create sdk types with low overhead Pull request opened by narape on <!date^1670318562^{date_short}|2022-12-06T09:22:42Z> TL;DR Extends SdkTypes to create

SdkType<T>

with low overhead Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Currently the way to create custom

SdkType<T>

in Java is using

JacksonSdkType

. For example If a developer wants to define a task equivalent to

(String str) -> str.toUpperCase()

currently he/she needs to write it as

@AutoService(SdkRunnableTask.class)
public class ToUpperCaseTask extends SdkRunnableTask<ToUpperCaseTask.Input, ToUpperCaseTask.Output> {
  public ToUpperCaseTask() {
    super(JacksonSdkType.of(Input.class), JacksonSdkType.of(Output.class));
  }

  @AutoValue
  public abstract static class Input {
    public abstract String in();
  }

  @AutoValue
  public abstract static class Output {
    public abstract String out();

    public static Output create(String out) {
      return new AutoValue_ToUpperCaseTask_Output(out);
    }
  }

  @Override
  public Output run(Input input) {
    return Output.create(input.toUpperCase()));
  }
}

For such trivial task, defining the AutoValues to define the types takes the majority of the lines. With this PR the developer could write the task this way instead

@AutoService(SdkRunnableTask.class)
public class ToUpperCaseTask extends SdkRunnableTask<String, String> {
  public ToUpperCaseTask() {
    super(SdkTypes.ofPrimitive("in", String.class), SdkTypes.ofPrimitive("out", String.class));
  }

  @Override
  public String run(String input) {
    return input.toUpperCase();
  }
}

Defining the types for the task now takes one line only and the

run

is simplified too as no need to create or access AutoValues This PR supports creating simplified `SdkType`s for the following types: •

SdkTypes.ofPrimitive("in", Long.class)

for

SdkType<Long>

•

SdkTypes.ofPrimitive("in", Double.class)

for

SdkType<Double>

•

SdkTypes.ofPrimitive("in", String.class)

for

SdkType<String>

•

SdkTypes.ofPrimitive("in", Boolean.class)

for

SdkType<Boolean>

•

SdkTypes.ofPrimitive("in", Instant.class)

for

SdkType<Instant>

•

SdkTypes.ofPrimitive("in", Duration.class)

for

SdkType<Duration>

Also support for collection and map for the the same primitive types: •

SdkTypes.ofCollection("in", Long.class)

for

SdkType<List<Long>>

•

SdkTypes.ofMap("in", Long.class)

for

SdkType<Map<String, Long>>

And finally support for struts, collection of structs and map of struts: •

SdkTypes.ofStruct("in", JacksonSdkType.of(Input.class))

for

SdkType<Input>

•

SdkTypes.ofCollection("in", JacksonSdkType.of(Input.class))

for

SdkType<List<Input>>

•

SdkTypes.ofMap("in", JacksonSdkType.of(Input.class))

for

SdkType<Map<String, Input>>

flyteorg/flytekit-java ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/5e2baee6f2ba0c12ddc123c7848438fa59f7a5a3|5e2baee6>

- chore(deps): bump ua-parser-js from 0.7.32 to 0.7.33 (#675) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/e843866da37e911753e24514c45abe18b4e79346|e843866d>

- chore(deps): bump http-cache-semantics from 4.1.0 to 4.1.1 (#678) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/92c8d9183e75963ba35ecac43633d8a20c40e3bf|92c8d918>

- [Snyk] Upgrade @typescript-eslint/eslint-plugin from 5.47.0 to 5.48.2 (#683) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/fcb41b524138b7c0b34f931f2e9827420f842667|fcb41b52>

- [Snyk] Upgrade @typescript-eslint/parser from 5.47.0 to 5.48.2 (#684) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/8a9386629fbcbb7292b54f53b90fa2c19c4d6f6d|8a938662>

- [Snyk] Upgrade prettier from 2.8.1 to 2.8.3 (#685) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/9325daa90f3ab1a19ab2c369c38b6f3f01643828|9325daa9>

- [Snyk] Upgrade @types/morgan from 1.9.3 to 1.9.4 (#686) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/131991703956894274ccf61272ffa4057358c430|13199170>

- [Snyk] Upgrade @types/react from 16.14.34 to 16.14.35 (#687) flyteorg/flyteconsole

Release - v1.4.0b1 New release published by eapolinario What's Changed • Support checkpointing in local mode from cached tasks by @bethebunny in #1457 • Deprecate FlyteSchema by @pingsutw in #1418 • Use scarf images by @eapolinario in #1434 • add undocumented objects/functions to flytekit api ref by @cosmicBboy in #1502 • Use non-root user in default flytekit image by @pingsutw in #1417 • Fix PyTorch transformer by @samhita-alla in #1510 • Fix mypy errors by @pingsutw in #1313 • Compile the workflow only at compile time by @pingsutw in #1311 • Get the origin type when serializing dataclass by @pingsutw in #1508 • Remove requirements files by @eapolinario in #1511 • Delay initialization of SynchronousFlyteClient in FlyteRemote by @eapolinario in #1514 • Make flytekit comply with PEP-561 by @pingsutw in #1516 • Flytekit Auth system overhaul and pretty printing upgrade by @kumare3 in #1458 • Create non-root user after apt-get by @eapolinario in #1519 • Add root pyflyte reference to docs by @eapolinario in #1520 • Drop support for python 3.7 by @eapolinario in #1521 • DuckDB plugin by @samhita-alla in #1419 Full Changelog: v1.3.2...v1.4.0b1 flyteorg/flytekit

#3377 [BUG] Readme link 404 error Issue created by r-matsuzaka Describe the bug I got the 404 errors for several links in Readme. Expected behavior I should see correct pages. Additional context to reproduce No response Screenshots These links does not work. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3378 [flytekit-java] Reduce verbosity when defining types with one variable Issue created by narape Motivation: Why do you think this is important? If a developer wants to define a task equivalent to (String str) -> str.toUpperCase() currently he/she needs to write it in Java as

@AutoService(SdkRunnableTask.class)
public class ToUpperCaseTask extends SdkRunnableTask<ToUpperCaseTask.Input, ToUpperCaseTask.Output> {
  public ToUpperCaseTask() {
    super(JacksonSdkType.of(Input.class), JacksonSdkType.of(Output.class));
  }

  @AutoValue
  public abstract static class Input {
    public abstract SdkBindingData<String> in();
  }

  @AutoValue
  public abstract static class Output {
    public abstract SdkBindingData<String> out();

    public static Output create(String out) {
      return new AutoValue_ToUpperCaseTask_Output(out);
    }
  }

  @Override
  public Output run(Input input) {
    return Output.create(SdkBindingDataFactory.of(input.get().toUpperCase()));
  }
}

For such trivial task, defining the AutoValues to define the types takes the majority of the lines, i.e. it is too verbose. It would be cool to have an alternative way to express the same in a simpler way Goal: What should the final outcome look like, ideally? I would like to have a way to define this simple types as one liner:

SdkTypes.of(SdkLiteralTypes.strings(), "str")

Describe alternatives you've considered I have considered code generation, but it has a cost and it is better to explore first alternatives without it Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/09dce86469eedd8da49da94feeb0297ab4196e20|09dce864>

- Literaltype as sdktype (#198) flyteorg/flytekit-java

#367 Create service for runtime metrics Pull request opened by hamersaw TL;DR This PR creates a gRPC service for exposing runtime metrics for workflows / nodes. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3272 Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/746c4a32f10ce339c7ec36709c403a732ccca155|746c4a32>

- fix: upgrade info.picocli:picocli from 4.7.0 to 4.7.1 (#197) flyteorg/flytekit-java

#307 Fixing pod plugin event reporting timestamps Pull request opened by hamersaw TL;DR Currently the pod plugin timestamps to detect when a pod started and finished are never hit. This means that when reporting these timestamps, Flyte uses the current time (ie. when Flyte checks the pod status) and consequently these are inaccurate. In this PR we attempt to use the timestamps from the container to ensure accurate reporting. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3272 Follow-up issue NA flyteorg/flyteplugins Codecov: 31.25% of diff hit (target 62.83%) Codecov: 62.75% (-0.09%) compared to f0db511 ✅ 4 other checks have passed 4/6 successful checks

#524 Add endpoint exposing runtime metrics Pull request opened by hamersaw TL;DR This PR implements an endpoint to expose runtime metrics from flyteadmin. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3272 Follow-up issue NA flyteorg/flyteadmin GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 8 other checks have passed 8/12 successful checks

#529 Add reported_at timestamps to events Pull request opened by hamersaw TL;DR This PR populates the

reported_at

field on workflow, node, and task events to more accurately track executions. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#3272 Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks