#1417 Use non-root user in default flytekit image Pull request opened by pingsutw Signed-off-by: Kevin Su pingsutw@apache.org Fixes flyteorg/flyte#3280 TL;DR Use non-root user in default flytekit image Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Create and update user in the docker file Tracking Issue flyteorg/flyte#3280 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3248 [charts/flyte-binary] Add provision for specifying configmaps for extra configs Pull request opened by jeevb • Adds a new value in

flyte-binary

chart for extra configuration to be passed in via an existing configmap (

inlineConfigMap

). If specified, this will be merged with the base configuration via a

projected

volume. • Leverages the above option in sandbox-bundled, and creates an empty configmap that can be updated by users to inject additional configuration and cluster resource templates for the flyte-sandbox. • Changed some values so that resources names are more concise in the sandbox. Updated the local single-binary flyte config example to reflect the new values. • Add secrets/webhook test to the

make test

target for

sandbox-bundled

. flyteorg/flyte ✅ All checks have passed 11/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/ec92a2c46e53954b1283a46f44e1826591cc246f|ec92a2c4>

- [charts/flyte-binary] Add provision for specifying configmaps for extra configs (#3248) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flytekit/commit/b6605bc95bd6f03434aeb5ca8f69eba161924ef8|b6605bc9>

- [Docs] SynchronousFlyteClient API reference #3095 (#1416) flyteorg/flytekit

#3095 [Docs] SynchronousFlyteClient API reference Issue created by samhita-alla Description Add

SynchronousFlyteClient

to Flytekit API reference guide. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#174 [Snyk] Upgrade com.google.errorprone:error_prone_annotations from 2.16 to 2.17.0 Pull request opened by snyk-bot Snyk has created this PR to upgrade com.google.errorprone:error_prone_annotations from 2.16 to 2.17.0. ℹ️ Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. * * * • The recommended version is 1 version ahead of your current version. • The recommended version was released 23 days ago, on 2022-12-30. * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs. For more information: https://camo.githubusercontent.com/7d4be2848d769770bfdfd3da24747fab9cae8633370b6c1d85bf5bbb7b38990e/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949355a4451314f444534597930794e474a6b4c5452684e5751744f5467334e5330795a5459315a4752695a6d51314e7a55694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a6c6b4e4455344d54686a4c544930596d51744e4745315a4330354f4463314c544a6c4e6a566b5a474a6d5a4455334e534a3966513d3d 🧐 View latest project report 🛠️ Adjust upgrade PR settings 🔕 Ignore this dependency or unsubscribe from future upgrade PRs flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

#3256 Added GPU enabled sandbox image. Pull request opened by ahlgol A new Dockerfile and build-target "build-gpu" in docker/sandbox-bundled that builds a CUDA enabled image named flyte-sandbox-gpu. Describe your changes • Build target added in Makefile for "build-gpu" that builds Dockerfile.gpu • Build target added in Makefile for "manifests-gpu" that adds gpu-operator.yaml to manifests • Dockerfile.gpu is based on existing Dockerfile, but uses a base image from nvidia and installs k3s and crictl and adds containerd config template for nvidia container runtime • Adds bin/k3d-entrypoint-gpu-check.sh that checks if container is started in an nvidia enabled image and exits otherwise. • bin/k3d-entrypoint.sh have been modified to allow for stderr to pass to output, so warning from other entrypoint scripts can be seen (now it will be missing in logfile however) Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Note to reviewers Changes have been added following info from these sources (plus some trial and error): https://itnext.io/enabling-nvidia-gpus-on-k3s-for-cuda-workloads-a11b96f967b0 https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html https://k3d.io/v5.4.6/usage/advanced/cuda/ flyteorg/flyte GitHub Actions: trigger-sandbox-bundled-build GitHub Actions: trigger-single-binary-build DCO: DCO ✅ 8 other checks have passed 8/11 successful checks

#3257 [BUG] [flytekit] Local execution of launch plan can be incorrect in some cases Issue created by kumare3 Describe the bug LaunchPlan's can contain - fixed_inputs and default_inputs. Fixed inputs are not overridable. But the way launchplans are handled in flytekit local execution today, users can accidentally override fixed inputs. this should fail in the backend compiler, but will not be caught in local execution this code sample shows how the saved inputs is a combination of the two and it can be seen here how this will not be dealt with correctly in local execution and even in workflow creation. Expected behavior Failure in local execution if a fixed input is overriden Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1418 Deprecate FlyteSchema Pull request opened by pingsutw Signed-off-by: Kevin Su pingsutw@apache.org fixes flyteorg/flyte#3278 TL;DR Plan to deprecate FlyteSchema in 1.4.0. After one year, we will remove it flyteorg/flytesnacks#942 Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue NA Follow-up issue NA flyteorg/flytekit Codecov: 69.45% (-0.01%) compared to 35b1fa6 Codecov: 0.00% of diff hit (target 69.45%) ✅ 28 other checks have passed 28/30 successful checks

#942 Deprecate FlyteSchema Pull request opened by pingsutw Signed-off-by: Kevin Su pingsutw@apache.org fixes: flyteorg/flyte#3278 flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#176 Bump errorprone and autovalue libs #patch Pull request opened by narape TL;DR Bump error prone to

2.17.0

and autovalue to

1.10.1

Type ☑︎ Bump dependencies ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Folds in the changes added by: • #174 • #160 Snyck doesn't sign its commits and also format the pom and incompatible way Tracking Issue NA Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by narape

<https://github.com/flyteorg/flytekit-java/commit/f38be8009e0bcf0dea0867e7a05198cff67e5534|f38be800>

- Bump errorprone and autovalue libs (#176) flyteorg/flytekit-java

#160 [Snyk] Upgrade com.google.auto.value:auto-value-annotations from 1.10 to 1.10.1 Pull request opened by EngHabu This PR was automatically created by Snyk using the credentials of a real user. Snyk has created this PR to upgrade com.google.auto.value:auto-value-annotations from 1.10 to 1.10.1. ℹ️ Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. * * * • The recommended version is 1 version ahead of your current version. • The recommended version was released a month ago, on 2022-11-17. * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs. For more information: https://camo.githubusercontent.com/19908c193e18f5aaa2c8589ba74273b1d85e55482b3aad94493e6c4bad849db5/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f6949315a44526c4f4449324e5330314d6d49314c5451304e54457459575178597930334f5459774e544934595441334d5455694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a566b4e4755344d6a59314c545579596a55744e4451314d5331685a44466a4c5463354e6a41314d6a68684d4463784e534a3966513d3d 🧐 View latest project report 🛠️ Adjust upgrade PR settings 🔕 Ignore this dependency or unsubscribe from future upgrade PRs flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytesnacks/tree/master|master>

by bstadlbauer

<https://github.com/flyteorg/flytesnacks/commit/2aefe89e3973f831412e476b1d6b4c422ad8768b|2aefe89e>

- Dask plugin docs (#929) flyteorg/flytesnacks

#929 Dask plugin docs Pull request opened by bstadlbauer Adds docs for the

dask

integration developed in flyteorg/flyte#427 Todo before merging (after flyteorg/flyteplugins#275 and flyteorg/flytekit#1366 are in): ☑︎ Revisit all links to make sure they work ☑︎ Add and lock

cookbook/integrations/kubernetes/k8s_dask/requirements.in

(on Linux) flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#1419 DuckDB plugin Pull request opened by samhita-alla Signed-off-by: Samhita Alla aallasamhita@gmail.com TL;DR This PR adds a

DuckDBQuery

task plugin that runs queries using DuckDB as the DBMS. Type ☐ Bug Fix ☐ Feature ☑︎ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description Capturing the crucial assumptions I made while building the task plugin: • The

DuckDBQuery

task parameter that a user needs to send argument to includes

query

and can contemplate adding includes

inputs

. •

query

can include a set of queries that'll be run sequentially. The last query needs to be a SELECT query. •

inputs

can include structured dataset or a list of parameters to be sent to the queries. • The

output

is a pyarrow table. Can be converted to any structured dataset compatible type. • The connection mode is set to

:memory

, i.e., the data is always stored in an in-memory, non-persistent database. It can be set to a file, but it's difficult to make the file accessible to different

DuckDBQuery

pods, which otherwise wouldn't make sense because file is persistent, and it needs to be leveraged. Example:

duckdb_query = DuckDBQuery(
    name="read_parquet",
    query=[
        "INSTALL httpfs",
        "LOAD httpfs",
        """SELECT hour(lpep_pickup_datetime) AS hour, count(*) AS count FROM READ_PARQUET(?) GROUP BY hour""",
    ],
    inputs=kwtypes(params=list[str]),
)

@workflow
def parquet_wf(parquet_file: str) -> pd.DataFrame:
    return duckdb_query(params=[parquet_file])

assert isinstance(
    parquet_wf(parquet_file="<https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-02.parquet>"),
    pd.DataFrame,
)

Tracking Issue Fixes flyteorg/flyte#3246 Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.10, flytekit-spark) GitHub Actions: build-plugins (3.10, flytekit-snowflake) GitHub Actions: build-plugins (3.10, flytekit-ray) GitHub Actions: build-plugins (3.10, flytekit-polars) GitHub Actions: build-plugins (3.10, flytekit-papermill) GitHub Actions: build-plugins (3.10, flytekit-pandera) GitHub Actions: build-plugins (3.10, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.10, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.10, flytekit-kf-mpi) GitHub Actions: build-plugins (3.10, flytekit-k8s-pod) GitHub Actions: build-plugins (3.10, flytekit-hive) GitHub Actions: build-plugins (3.10, flytekit-duckdb) GitHub Actions: build-plugins (3.10, flytekit-dolt) GitHub Actions: build-plugins (3.10, flytekit-deck-standard) GitHub Actions: build-plugins (3.10, flytekit-dbt) GitHub Actions: build-plugins (3.10, flytekit-data-fsspec) GitHub Actions: build-plugins (3.10, flytekit-dask) GitHub Actions: build-plugins (3.10, flytekit-bigquery) GitHub Actions: build-plugins (3.10, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.10, flytekit-aws-batch) GitHub Actions: build-plugins (3.10, flytekit-aws-athena) GitHub Actions: build-plugins (3.9, flytekit-whylogs) GitHub Actions: build-plugins (3.9, flytekit-vaex) GitHub Actions: build-plugins (3.9, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.9, flytekit-spark) GitHub Actions: build-plugins (3.9, flytekit-snowflake) GitHub Actions: build-plugins (3.9, flytekit-ray) ✅ 1 other check has passed 1/30 successful checks

Pending review on flyteorg/flytestdlib - Manage reminder [#117] Update boilerplate version (flyte-bot) 6 months stale · a year old · Waiting on _ @Haytham Abuelfutuh, katrogan, @Yee, @Ketan (kumare3)_

#1420 Auto Backfill workflow Pull request opened by kumare3 TL;DR This PR introduces a new command called

pyflyte backfill

and equivalent

remote.launch_backfill

that can create a backfiller workflow for a given launchplan automatically and register it statically Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Issue fixes flyteorg/flyte#3212 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#943 Add dbt example Pull request opened by eapolinario This PR adds a self-contained dbt example that runs on the sandbox. We need some wordsmithing to make sure the example makes sense and can be followed./ fixes flyteorg/flyte#2965 flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#1421 Sqlalchemy multiline query Pull request opened by cosmicBboy TL;DR There's a bug in

SQLAlchemyTask

where multi-line strings were not valid inputs. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Like the

SQLite3Task

, modify

SQLAlchemyTask

to sanitize the input string. Tracking Issue NA Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flyteplugins/commit/4634a81403e501882e3b1f39bcbd229b78768a4e|4634a814>

- Add env FLYTE_FAIL_ON_ERROR to aws batch job (#306) flyteorg/flyteplugins

Release - v1.0.31 New release published by flyte-bot Changelog • 4634a81 Add env FLYTE_FAIL_ON_ERROR to aws batch job (#306) flyteorg/flyteplugins

#944 Revamp Getting Started Section in Flytesnacks Pull request opened by cosmicBboy Fixes flyteorg/flyte#3240 This PR adds new content to the Getting Started section of the Flyte documentation: The current Getting Started page is in the

flyte

repo. This PR moves that to the flytesnacks repo, where flyteorg/furo#22 needs to be merged in order for the theme updates to take effect. The following subsections have been added: • Flyte Fundamentals: basically this is a gateway into the User Guide section, introducing newcomers to Flyte's main concepts and development lifecycle • Core Use Cases: this is a gateway to the Tutorial section, introducing data, ML, and analytics use cases. flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#674 chore: cleanup packages Pull request opened by ursucarina Signed-off-by: Carina Ursu carina@union.ai TL;DR Consolidates imported packages Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 8 other checks have passed 8/11 successful checks

Release - flytesnacks v0.3.161 New release published by flyte-bot Changelog • 2aefe89 Dask plugin docs (#929) • 9a1b05a Minor typo in Golang (#941) • 9e28039 Document IgnoreOutputs exception (#938) flyteorg/flytesnacks

#3034 [Core feature] Deprecate CreateDownloadLocation API to allow for a more controllable CreateDownloadLink API. Issue created by EngHabu Motivation: Why do you think this is important? There are a few issues with the current API: • Admin is unable to link the request to an actual execution's output. It's, therefore, unable to enforce any authorization rules. • It exposes the entirety of the s3 bucket admin has access to, to be downloaded through the API with no meaningful Audit logs to record. • It assumes the deck/output file locations are known and not changeable which, I believe, isn't the right assumption. Ideally, even for outputs, users can just request the output location through the API. This makes the actual storage behind the output/decks/metadata completely portable. Goal: What should the final outcome look like, ideally? The request should probably change from:

// CreateDownloadLocationRequest specified request for the CreateDownloadLocation API.
message CreateDownloadLocationRequest {
  // NativeUrl specifies the url in the format of the configured storage provider (e.g. <s3://my-bucket/randomstring/suffix.tar>)
  string native_url = 1;
  // ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this
  // exceeds the platform allowed max.
  // +optional. The default value comes from a global config.
  google.protobuf.Duration expires_in = 2;
}

to something like:

message CreateDownloadLinkRequest {
  oneof source {
    // WorkflowId is the unique identifier for the workflow execution.
    core.WorkflowExecutionIdentifier execution_id = 1;

    // NodeId is the unique identifier for the node execution. For a task node, this will retrieve the output of the
    // most recent attempt of the task.
    core.NodeExecutionIdentifier node_id = 2;

    // TaskId is the unique task execution identifier that points to a specific attempt of the task.
    core.TaskExecutionIdentifier task_id = 3;
  }

  ArtifactType artifact_type = 4;

  // ExpiresIn defines a requested expiration duration for the generated url. The request will be rejected if this
  // exceeds the platform allowed max.
  // +optional. The default value comes from a global config.
  google.protobuf.Duration expires_in = 5;
}

Describe alternatives you've considered Admin parsing the request URL and mapping to an execution: puts a lot of assumptions on the schema of the URL. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3258 [flytekit-java] Add toSdkBindingDataMap() to SdkType Issue created by andresgomezfrr Motivation: Why do you think this is important? Add a new method to the SdkType converters to transform an input/output class in a

Map<String, SdkBindingData<?>>

. This feature will be used as a part of the typed input implementation, which will be implemented in other PR. Goal: What should the final outcome look like, ideally? The goal is to have a method that allows you to transform an input/output class in a Map<String, SdkBindingData>` where the key is the attribute names and the values are the SdkBindingData value of the inputs/outputs. Describe alternatives you've considered no alternatives Propose: Link/Inline OR Additional context flyteorg/flytekit-java#177 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#177 Add toSdkBindingDataMap() to SdkType Pull request opened by andresgomezfrr TL;DR Add a new method into the SdkType interface to allow to transform of an input/output class in a

Map<String, SdkBindingData<?>>

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Add a new method into the SdkType interface to allow to transform an input/output class in a Map<String, SdkBindingData<?>> This PR also adds their implementations (JacksonSdkType, SdkScalaType) Tracking Issue • flyteorg/flyte#3258 Follow-up issue NA flyteorg/flytekit-java ✅ All checks have passed 3/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit-java/tree/master|master>

by andresgomezfrr

<https://github.com/flyteorg/flytekit-java/commit/57ad4540348bc2ef487cfd4be8d78beb5d9503fb|57ad4540>

- Add toSdkBindingDataMap() to SdkType (#177) flyteorg/flytekit-java

#3259 [Docs] Fix dask integration page TOC Issue created by samhita-alla Description Dask integration page is hidden in the documentation. Will need to update the TOC to make it visible on the integrations page. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte