#3804 [Core feature] Add LaunchPlan Archive/Activate support via flytekit Issue created by samjeckert Motivation: Why do you think this is important? We're able to move most control functionality that was on flytectl to flytekit, but we are missing the capability for activating/archiving existing launchplans. To solve this, we use a hybrid system of both CLI and flytekit, which is a bit unwieldy. We're mostly concerned with managing active crons. Goal: What should the final outcome look like, ideally? • Ability to get all launchplan versions for a given lp name • Ability to deprecate a launchplan • Ability to activate a launchplan Describe alternatives you've considered We use calls out to

flytectl

to fetchall, archive, and activate launchplans Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/f8a82b5c60b323a123b458111ec61f01ca49f1d6|f8a82b5c>

- Fail registration if output isn't Optional when using map tasks with min_success_ratio < 1 (#1709) flyteorg/flytekit

#22 Add auth arch diagram Pull request opened by davidmirror-ops Add diagram to be referenced on an updated version of the "Authenticanting on Flyte" page. flyteorg/static-resources ✅ All checks have passed 2/2 successful checks

#3805 [BUG] Support client secret env var using flytectl config in flytekit Issue created by pmahindrakar-oss Describe the bug Currently we have to explicity set FLYTE_CREDENTIALS_CLIENT_SECRET inorder for flytekit remote to use the client secret and doesn't use the env variable defined in the flytectl config.yaml eg :

root:
  project: flytesnacks
  domain: development
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte-endpoint
  authType: ClientSecret
  clientId: sample-client-id
  clientSecretEnvVar: sample_client_secret
  insecure: false
logger:
  show-source: true
  level: 0

Expected behavior flyte remote should be able to read flytectl environment variable for client secret Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#390 Add ArrayNode proto message Pull request opened by hamersaw TL;DR This PR adds the

ArrayNode

definition as a new node type in Flyte. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue flyteorg/flyte#1131 Follow-up issue NA flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

#418 Backport Flyte 1.7 Pull request opened by eapolinario Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl DCO: DCO ✅ 22 other checks have passed 22/23 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/0b64f21d68a2c8f6dd6f9f1d833ad28aabe5d674|0b64f21d>

- Add bigquery project and location to metadata (#1702) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/cae30389d1cec45c0eefa283a4f32b5707db229b|cae30389>

- Add single file notes flyteorg/community

#12 Add 06/13/23 meeting notes Pull request opened by davidmirror-ops flyteorg/community DCO: DCO ✅ 1 other check has passed 1/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/810f78714225295ee014ac0a426e5985ac583525|810f7871>

- Add 06/13/23 meeting notes (#12) flyteorg/community

#13 Add contributors meetup notes Pull request opened by davidmirror-ops flyteorg/community ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/8de9d7693bf53b680ccc43448ef03a5580299ae6|8de9d769>

- Add contributors meetup notes (#13) flyteorg/community

#3806 [Housekeeping] Inconsistent treatment of project id param in flytectl Issue created by katrogan Describe the issue

flytectl update project --help

shows

id

as a flag:

Flags:
...
      --id string               id for the project specified as argument.

However attempting to update a project with the id flag fails:

flytectl update project --id katrina-test                                                                                                                                                  <region:us-west-2>
Error: project id wasn't passed

and requires using the global

-p

flag instead What if we do not do this? Poor DevX using the project noun in flytectl Related component(s) flytectl Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#14 Add contributors meetup notes - 06/22/23 Pull request opened by davidmirror-ops flyteorg/community ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/a6ba43a5f6cc99780d85821c686af2875b284dec|a6ba43a5>

- Add contributors meetup notes - 06/22/23 (#14) flyteorg/community

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/3b064dca90de2d0d2d691e931aaadca56beb3a7f|3b064dca>

- Generate decks at local execution (#1699) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/dcce1a45cdc91799329f96eba1015809667557df|dcce1a45>

- Add contributors meetup flyteorg/community

#782 chore: export storage utils Pull request opened by ursucarina Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteconsole ✅ All checks have passed 2/2 successful checks

#3808 Fix single binary config Pull request opened by Yicheng-Lu-llll Describe your changes The existing flyte_local.yaml configuration file has fallen out of date, causing issues when running single binaries. The standard process ideally looks like this:

# Start a k3s cluster, create pods for postgres, minio.
flytectl demo start --dev
# Start all backends (flyteidl, flyteadmin, flyteplugins, flytepropeller) and HTTP Server in a single binary 
sudo make compile
flyte start --config flyte_local.yaml

This PR introduces necessary updates to the flyte_local.yaml configuration file. Checks

flytectl demo start --dev
# 👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉 
# ❇️ Run the following command to export demo environment variables for accessing flytectl
#         export FLYTECTL_CONFIG=/root/.flyte/config-sandbox.yaml 
# 🐋 Flyte sandbox ships with a Docker registry. Tag and push custom workflow images to localhost:30000
# 📂 The Minio API is hosted on localhost:30002. Use <http://localhost:30080/minio/login> for Minio console

sudo make compile
flyte start --config flyte_local.yaml

# running a hello_world sample from flytesnacks
pyflyte run --remote core/flyte_basics/hello_world.py my_wf
# Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f40d94cefb5e74fe6b09> to see execution in the console.

pyflyte run --remote core/flyte_basics/deck.py wf
# Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f39a9c56828e54bb09be> to see execution in the console.

Able to run the task Able to see the deck: Able to see the kubernetes logs: flyteorg/flyte ✅ All checks have passed 6/6 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flyteplugins/commit/1d0f3e9d9b5264459c97e6fb2bc64bc0147971d1|1d0f3e9d>

- Pass location to the BigQuery request (#365) flyteorg/flyteplugins

Release - v1.1.3 New release published by flyte-bot Changelog • 1d0f3e9 Pass location to the BigQuery request (#365) flyteorg/flyteplugins

#366 Switch to official dask operator Pull request opened by bstadlbauer TL;DR This replaces the

go

dask

client from

<http://github.com/bstadlbauer/dask-k8s-operator-go-client|github.com/bstadlbauer/dask-k8s-operator-go-client>

with the official one from

<http://github.com/dask/dask-kubernetes/v2023|github.com/dask/dask-kubernetes/v2023>

. The operator has been moved over from my repo to the official one in dask/dask-kubernetes#731, so the code is essentially the same, but I'm going ahead and will archive my version. Type ☐ Bug Fix ☐ Feature ☑︎ Plugin flyteorg/flyteplugins Codecov: 62.75% (-1.31%) compared to dfdf6f9 ✅ 6 other checks have passed 6/7 successful checks

#3809 [BUG] Notebook task run as a AWS Map Task throw exception Issue created by jw0515 Describe the bug

import os
import pathlib

from flytekitplugins.papermill import NotebookTask
from flytekitplugins.awsbatch import AWSBatchConfig

config = AWSBatchConfig(
    platformCapabilities="EC2",
)

backtest_wages = NotebookTask(
    name="test-notebook",
    notebook_path=os.path.join(
        pathlib.Path(__file__).parent.parent.absolute(), "notebooks/test-notebook.ipynb"
    ),
    render_deck=True,
    task_config=config
)

Tried to use AWS batch job to run the notebook task using the code example above, got an error:

logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'

When I looked at the batch job docker command:

["pyflyte-execute","--inputs","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/inputs.pb>","--output-prefix","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/0/0>","--raw-output-data-prefix","<s3://fiolito-eks-cluster/99/alg58fr86fmkgh9nqnf4-n0-0>","--resolver","flytekit.core.python_auto_container.default_task_resolver","--","task-module","flytekitplugins.papermill.task","task-name","_dummy_task_func"]

It seems that the _dummy_task_func is wrong which should be

test-notebook

and it comes from: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L145 Expected behavior Notebook task can run as an AWS Map Task without throwing exception. Additional context to reproduce Python3.10 flytekit 1.7.0 flytekitplugin-awsbatch 1.7.0 flytekitplugin-papermil 1.7.0 Screenshots in aws batch console: Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/5a5ad821b7ba3c51166a40c11d1695fb79a7ea8d|5a5ad821>

- Fix map task cache misses (#363) flyteorg/flyteplugins

Release - v1.1.4 New release published by flyte-bot Changelog • 5a5ad82 Fix map task cache misses (#363) flyteorg/flyteplugins

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/2c393c030f44a87a4b67a859ffc6059fd8de50fe|2c393c03>

- Switch to official dask operator (#366) flyteorg/flyteplugins

Release - v1.1.5 New release published by flyte-bot Changelog • 2c393c0 Switch to official dask operator (#366) flyteorg/flyteplugins

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/1052af8c1a4eff426023db19e1fba4441a03e490|1052af8c>

- Improve task type hint (#1711) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteidl/commit/b219c2ab37886801039fda67d913760ac6fc4c8b|b219c2ab>

- Add ArrayNode proto message (#390) flyteorg/flyteidl

#1712 Async Agent Pull request opened by pingsutw TL;DR Use async API in the agent service Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytekit