Release - v1.0.42 New release published by flyte-bot Changelog • 24d8aba enabling connection pooling (#89) flyteorg/datacatalog

#3269 Docs update to clarify rawContainer tasks and relationship Issue created by kumare3 Description The below statement needs to be added to the Raw Containers page:

One of the besnefit of raw container tasks is that one does not need to
install flytekit in the target container.

Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytekit/commit/7b48fff5cfe2fcf19e28fde8e35c20acdb9059e2|7b48fff5>

- Sqlalchemy multiline query (#1421) flyteorg/flytekit

#311 [WIP] Out of core plugin Pull request opened by pingsutw Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteplugins GitHub Actions: Run tests and lint GitHub Actions: generate WIP: Title contains "WIP" ✅ 1 other check has passed 1/4 successful checks

#427 [Backend][Plugin]Support for Dask clustered tasks in Flyte Issue created by kumare3 Why would this plugin be helpful to the Flyte community Users could write very short running distributed array jobs using DASK. This makes it possible to have very small runtime jobs multi-plexed onto same set of nodes. Type of Plugin ☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...) ☑︎ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...) ☐ Customized Plugin using native kubernetes constructs ☐ Other Can you help us with the implementation? ☐ Yes ☐ No Additional context This would really help express some ideas that are not Spark, or heavyweight like Flyte batch jobs. flyteorg/flyte

#3271 [Docs] Add new signaling functions/classes to API reference and flytesnacks example Issue created by cosmicBboy Description The

wait_for_input

,

approve

and

sleep

functions to use gate nodes in flytekit workflows need to be added to the API reference. • Add these functions to the module-level docstring for flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3274 [Core feature] Observability: benchmarking - create an objective set of benchmarks so we can see perf improvements/regressions on some different set of Flyte cluster configs Issue created by cosmicBboy Motivation: Why do you think this is important? Blocked by: #3272, #3273 [to be filled out] Goal: What should the final outcome look like, ideally? [to be filled out] Describe alternatives you've considered [to be filled out] Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3275 Create Flyte org stats dashboard Issue created by cosmicBboy The purpose of this task is to create a dashboard similar to this, which shows some of the key metrics aggregated across all the officially-maintained Flyte oss repos. flyteorg/flyte

#3276 [Core feature] Migrate golang repos to monorepo Issue created by cosmicBboy Motivation: Why do you think this is important? [to be filled out] Goal: What should the final outcome look like, ideally? [to be filled out] Describe alternatives you've considered [to be filled out] Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3277 [Core feature] Add support for MySQL as a supported database for flyteadmin, datacatalog Issue created by cosmicBboy Motivation: Why do you think this is important? [to be filled out] Goal: What should the final outcome look like, ideally? [to be filled out] Describe alternatives you've considered [to be filled out] Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3278 [Core feature] Deprecation messaging - Deprecate FlyteSchema 12 releases from now Issue created by cosmicBboy Motivation: Why do you think this is important? [to be filled out] Goal: What should the final outcome look like, ideally? [to be filled out] Describe alternatives you've considered [to be filled out] Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3129 [BUG] Uploading workflows during workflow tutorial has issues when running locally. Issue created by kannon92 Describe the bug When following https://github.com/flyteorg/flyte-conference-talks/tree/main/pydata-global-2022 for the local setup, I am getting grpc connection refused when uploading workflows. I am able to run the example workflow using python but I can't upload packages to my local flyte server.

(flyte) Kevins-MBP:pydata-global-2022 kevinhannon$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v0
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
| NAME                                                                | STATUS | ADDITIONAL INFO                                       |
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
| /tmp/register3914406486/00_workflows.example_00_intro.get_data_1.pb | Failed | Error registering file due to rpc error: code =       |
|                                                                     |        | Unavailable desc = connection error: desc =           |
|                                                                     |        | "transport: Error while dialing dial tcp [::1]:30081: |
|                                                                     |        | connect: connection refused"                          |
 --------------------------------------------------------------------- -------- ------------------------------------------------------- 
1 rows
Error: Connection Info: [Endpoint: localhost:30081, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp [::1]:30081: connect: connection refused"
{"json":{"src":"main.go:13"},"level":"error","msg":"Connection Info: [Endpoint: localhost:30081, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [::1]:30081: connect: connection refused\"","ts":"2022-12-03T11:02:04-05:00"}

Expected behavior I expect that my workflows are uploaded and I can view them in the UI. Additional context to reproduce

uname -a

Darwin Kevins-MBP 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 004726 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T8101 arm64 Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3280 [Core feature] do not use root user in default flytekit image Issue created by cosmicBboy Motivation: Why do you think this is important? TODO Goal: What should the final outcome look like, ideally? TODO Describe alternatives you've considered TODO Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3281 [BUG] Failed to abort Databricks job Issue created by cosmicBboy Describe the bug It's related to this issue propeller panic because taskCtx.ResourceMeta() is nil when running databricks job However, it should never happen. it was happening because we tried to abort the task that doesn't exist. Expected behavior Users should be able to successfully delete the job Additional context to reproduce 1. Run a databricks job without access token 2. Try to abort the job Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3282 [Core feature] External WebAPI plugins: write plugins with FastAPI server without having to compile Issue created by cosmicBboy Motivation: Why do you think this is important? TODO Goal: What should the final outcome look like, ideally? TODO Describe alternatives you've considered TODO Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3283 [Core feature] Create reference implementation for Flyte deployments Issue created by cosmicBboy Motivation: Why do you think this is important? The purpose if this issue is to create reference implementations of terraform for deploying Flyte on AWS and GCP (eventually we may have community contributions for other cloud platforms like Azure). This reference implementation will be a useful starting point for Flyte administrators to start their production Flyte journey. Goal: What should the final outcome look like, ideally? By the end of this, we should have • terraform artifacts for deploying a Flyte cluster to AWS/GCP • documentation in the Deployment Guides that describe and take users through how to set it up. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3284 [Housekeeping] Improve the way requirements are regenerated Issue created by cosmicBboy Describe the issue TODO regenerating requirements: • 1 requirements files per python version? • tox? • dockerize? • how to manage plugins requirements? What if we do not do this? TODO Related component(s) flytekit Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3285 [Docs] vaex integration: create flytesnacks example page Issue created by cosmicBboy Description The Vaex integration currently does not have an example page under the integrations section: https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/flytekit_plugins The purpose of this task is to add (a) a page describing the plugin and how to install it (see here) and (b) an example page on how to use it (see here) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3286 [Docs] mlflow integration: create flytesnacks example page Issue created by cosmicBboy Description The MLFlow integration currently does not have an example page under the integrations section: https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/flytekit_plugins The purpose of this task is to add (a) a page describing the plugin and how to install it (see here) and (b) an example page on how to use it (see here) Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3287 [Docs] Update flytekit API reference configuration page: include flytectl config.yaml as source of configuration Issue created by cosmicBboy Description The configuration api reference does not mention that configuration for flytekit can come from the flytectl

config.yaml

file. Will need to update the docs to explain this. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1425 Add `pod_template` and `pod_template_name` arguments for `PythonAutoContainerTask`, its downstream tasks, and `@task`. Pull request opened by ByronHsu TL;DR Add

pod_template

and

pod_template_name

arguments for

PythonAutoContainerTask

, its downstream tasks, and

@task

. (Following this guide) Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description 1. Users can pass in

pod_template

in

PythonAutoContainerTask

, its downstream tasks, and

@task

to customize

pod_template

for each task instance. It also allows developers to set the default

pod_template

for plugins in flytekit. 2. Users can pass in

pod_template_name

in

PythonAutoContainerTask

, its downstream tasks, and

@task

to select which

PodTemplate

resources to use on the server side. Example • instance

task_with_minimum_pod_template = DummyAutoContainerTask(
    name="x",
    task_config=None,
    task_type="t",
    container_image="repo/image:0.0.0",
    pod_template=PodTemplate(
        primary_container_name="primary",
        labels={"lKeyA": "lValA"},
        annotations={"aKeyA": "aValA"},
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                ),
            ]
        ),
    ),
    pod_template_name="A",
)

• decorator

@task(
        container_image="repo/image:0.0.0",
        pod_template=PodTemplate(
            primary_container_name="primary",
            labels={"lKeyA": "lValA"},
            annotations={"aKeyA": "aValA"},
            pod_spec=V1PodSpec(
                containers=[
                    V1Container(
                        name="primary",
                    ),
                ]
            ),
        ),
        pod_template_name="A",
    )
    def func_with_pod_template(i: str):
        print(i + 3)

Tracking Issue flyteorg/flyte#3123 flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3288 [BUG] (Auth) Device flow token expiry not set correctly Issue created by hajapy Describe the bug When using the deviceflow with an external auth server, flytectl repeats the device flow each time it is used because it thinks the token is expired. Tracing through the code, I found this is probably the cause. The api response schema, notably has an

expires_in

not an

expiry

field. When unmarshalling to an oauth2.Token, it omits the missing

expiry

and sets the expiry to

0001-01-01T00:00:00Z

, which you can verify by looking in your keyring

keyring get flytectl 'dns:///<your-endpoint>:flytectl-user'

. This causes the future invocations of flytectl to think the token is expired and restart the device flow. Expected behavior The stored token should have the expiry date computed from the optional

expires_in

response from the /token api. Additional context to reproduce 1. Configure flytectl to use

authType: DeviceFlow

and point to an endpoint that has auth enabled 2. Run

flytectl version

3. Complete device auth flow 4. Run

flytectl version

again, notice you need to redo the device auth flow 5. Inspect contents of keyring and note the expiry date is the zero value of

time.Time

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3289 [BUG] Archive LaunchPlan Issue created by james-union Describe the bug We need to be able to archive launch plans as well as workflows or tasks. Currently, we don't have ability to archive launch plans on the backend side. Here, The following

PUT

method for

<https://development.uniondemo.run/api/v1/named_entities/3>

fails with the error

4Resource [LAUNCH_PLAN] cannot have its state updated4Resource [LAUNCH_PLAN] cannot have its state updated

Expected behavior I expect the API should update the state of Launch Plan successfully so that we can implement archiving launch plans on the UI side to follow this UI ticket. flyteorg/flyteconsole#665 Additional context to reproduce The discussion was held on this thread https://unionai.slack.com/archives/C02ALNHFCB1/p1674664981661909 Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#946 fix file name to enable appearance in left nav Pull request opened by ppiegaze New pull request using different branch name (not master) flyteorg/flytesnacks GitHub Actions: Mark github pre-release as Release GitHub Actions: Publish artifacts to github release GitHub Actions: Create Prerelease GitHub Actions: Bump Version ✅ 26 other checks have passed 26/30 successful checks

#3290 [FlyteCTL Feature] flytectl not recognized as a valid command on Windows Issue created by davidmirror-ops Describe the feature/command for FlyteCTL After following the documented instructions to install flytectl on Windows, the process completes successfully but, when the command is invoked, it returns an error:

flytectl' is not recognized as an internal or external command, operable program or batch file

Alternatives tried with no success: 1. Invoking the command using the full path (

C:\Users\<user>\bin\flytectl version

or

./bin/flytectl version

) Environment Windows 10 Pro, Dev version, ARM-64 arch, WSL2 with Ubuntu distro Provide a possible output or UX example Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/1513773eed118fc3ddacffddf71ee3d30cd0c5cf|1513773e>

- Sklearn type transformer should be automatically loaded with import flytekit (#1423) flyteorg/flytekit

#675 chore(deps): bump ua-parser-js from 0.7.32 to 0.7.33 Pull request opened by dependabot[bot] Bumps ua-parser-js from 0.7.32 to 0.7.33. Changelog Sourced from ua-parser-js's changelog.

Version 0.7.32 / 1.0.32

• Add new browser : DuckDuckGo, Huawei Browser, LinkedIn

• Add new OS : HarmonyOS

• Add some Huawei models

• Add Sharp Aquos TV

• Improve detection Xiaomi Mi CC9

• Fix Sony Xperia 1 III misidentified as Acer tablet

• Fix Detect Sony BRAVIA as SmartTV

• Fix Detect Xiaomi Mi TV as SmartTV

• Fix Detect Galaxy Tab S8 as tablet

• Fix WeGame mistakenly identified as WeChat

• Fix included commas in Safari / Mobile Safari version

• Increase UA_MAX_LENGTH to 350

Version 0.7.33 / 1.0.33

• Add new browser : Cobalt

• Identify Macintosh as an Apple device

• Fix ReDoS vulnerability

Version 0.8

Version 0.8 was created by accident. This version is now deprecated and no longer maintained, please update to version 0.7 / 1.0.

Commits • `f2d0db0` Bump version 0.7.33 • `a6140a1` Remove unsafe regex in trim() function • `a886604` Fix #605 - Identify Macintosh as Apple device • `b814bcd` Merge pull request #606 from rileyjshaw/patch-1 • `7f71024` Fix documentation • `c239ac5` Merge pull request #604 from obecerra3/master • `8d3c2d3` Add new browser: Cobalt • See full diff in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting

@dependabot rebase

. * * * Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: •

@dependabot rebase

will rebase this PR •

@dependabot recreate

will recreate this PR, overwriting any edits that have been made to it •

@dependabot merge

will merge this PR after your CI passes on it •

@dependabot squash and merge

will squash and merge this PR after your CI passes on it •

@dependabot cancel merge

will cancel a previously requested merge and block automerging •

@dependabot reopen

will reopen this PR if it is closed •

@dependabot close

will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually •

@dependabot ignore this major version

will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this minor version

will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) •

@dependabot ignore this dependency

will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) •

@dependabot use these labels

will set the current labels as the default for future PRs for this repo and language •

@dependabot use these reviewers

will set the current reviewers as the default for future PRs for this repo and language •

@dependabot use these assignees

will set the current assignees as the default for future PRs for this repo and language •

@dependabot use this milestone

will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the Security Alerts page. flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release GitHub Actions: Build Flyteconsole Image / Push to Github Registry ✅ 7 other checks have passed 7/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by jeevb

<https://github.com/flyteorg/flyte/commit/c4e4872824e96192af7a2aa569c050b78ab74332|c4e48728>

- Update flyte-sandbox subcharts and add support for data persistence a… (#3266) flyteorg/flyte

#3291 [Core feature] Map tasks should have the option to use the same logging config as that defined for tasks Issue created by rupshac Motivation: Why do you think this is important? I've recently started using map tasks and realized that I need to redefine the logging template that was already defined for tasks, without which I can't get to the logs of map tasks. This change is also not clearly documented hence came as a surprise. Goal: What should the final outcome look like, ideally? Ideally map tasks should use the same logging config as that for tasks by default. Users should have the option to override if they so desire. Describe alternatives you've considered I've tried @hamersaw 's recommendation from this thread and it's cumbersome Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3293 [BUG] Consolidate default log formats in flytekit and the backend Issue created by eapolinario Describe the bug We currently use 2 slightly different logging formats in flytekit and the backend components. This forces users to maintain multiple parsers to be able to ingest Flyte logs. Expected behavior We need to be backwards compatible, so at the very least we should expose the logging format used by flytekit. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte