GitHub
06/26/2023, 5:21 PMflytectl
to fetchall, archive, and activate launchplans
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/26/2023, 5:45 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flytekit/commit/f8a82b5c60b323a123b458111ec61f01ca49f1d6|f8a82b5c>
- Fail registration if output isn't Optional when using map tasks with min_success_ratio < 1 (#1709)
flyteorg/flytekitGitHub
06/26/2023, 5:51 PMGitHub
06/26/2023, 7:07 PMroot:
project: flytesnacks
domain: development
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///flyte-endpoint
authType: ClientSecret
clientId: sample-client-id
clientSecretEnvVar: sample_client_secret
insecure: false
logger:
show-source: true
level: 0
Expected behavior
flyte remote should be able to read flytectl environment variable for client secret
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/26/2023, 7:35 PMArrayNode
definition as a new node type in Flyte.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☐ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
flyteorg/flyte#1131
Follow-up issue
NA
flyteorg/flyteidl
✅ All checks have passed
13/13 successful checksGitHub
06/26/2023, 7:44 PMGitHub
06/26/2023, 7:52 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/0b64f21d68a2c8f6dd6f9f1d833ad28aabe5d674|0b64f21d>
- Add bigquery project and location to metadata (#1702)
flyteorg/flytekitGitHub
06/26/2023, 8:31 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/cae30389d1cec45c0eefa283a4f32b5707db229b|cae30389>
- Add single file notes
flyteorg/communityGitHub
06/26/2023, 8:38 PMGitHub
06/26/2023, 8:40 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/810f78714225295ee014ac0a426e5985ac583525|810f7871>
- Add 06/13/23 meeting notes (#12)
flyteorg/communityGitHub
06/26/2023, 8:43 PMGitHub
06/26/2023, 8:44 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/8de9d7693bf53b680ccc43448ef03a5580299ae6|8de9d769>
- Add contributors meetup notes (#13)
flyteorg/communityGitHub
06/26/2023, 8:45 PMflytectl update project --help
shows id
as a flag:
Flags:
...
--id string id for the project specified as argument.
However attempting to update a project with the id flag fails:
flytectl update project --id katrina-test <region:us-west-2>
Error: project id wasn't passed
and requires using the global -p
flag instead
What if we do not do this?
Poor DevX using the project noun in flytectl
Related component(s)
flytectl
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
06/26/2023, 8:45 PMGitHub
06/26/2023, 8:46 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/a6ba43a5f6cc99780d85821c686af2875b284dec|a6ba43a5>
- Add contributors meetup notes - 06/22/23 (#14)
flyteorg/communityGitHub
06/26/2023, 8:57 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/3b064dca90de2d0d2d691e931aaadca56beb3a7f|3b064dca>
- Generate decks at local execution (#1699)
flyteorg/flytekitGitHub
06/26/2023, 9:02 PM<https://github.com/flyteorg/community/tree/main|main>
by davidmirror-ops
<https://github.com/flyteorg/community/commit/dcce1a45cdc91799329f96eba1015809667557df|dcce1a45>
- Add contributors meetup
flyteorg/communityGitHub
06/26/2023, 9:49 PMGitHub
06/26/2023, 10:26 PM# Start a k3s cluster, create pods for postgres, minio.
flytectl demo start --dev
# Start all backends (flyteidl, flyteadmin, flyteplugins, flytepropeller) and HTTP Server in a single binary
sudo make compile
flyte start --config flyte_local.yaml
This PR introduces necessary updates to the flyte_local.yaml configuration file.
Checks
flytectl demo start --dev
# 👨💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
# ❇️ Run the following command to export demo environment variables for accessing flytectl
# export FLYTECTL_CONFIG=/root/.flyte/config-sandbox.yaml
# 🐋 Flyte sandbox ships with a Docker registry. Tag and push custom workflow images to localhost:30000
# 📂 The Minio API is hosted on localhost:30002. Use <http://localhost:30080/minio/login> for Minio console
sudo make compile
flyte start --config flyte_local.yaml
# running a hello_world sample from flytesnacks
pyflyte run --remote core/flyte_basics/hello_world.py my_wf
# Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f40d94cefb5e74fe6b09> to see execution in the console.
pyflyte run --remote core/flyte_basics/deck.py wf
# Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f39a9c56828e54bb09be> to see execution in the console.
Able to run the task
image▾
image▾
image▾
GitHub
06/27/2023, 3:57 AM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flyteplugins/commit/1d0f3e9d9b5264459c97e6fb2bc64bc0147971d1|1d0f3e9d>
- Pass location to the BigQuery request (#365)
flyteorg/flytepluginsGitHub
06/27/2023, 3:58 AMGitHub
06/27/2023, 8:19 AMgo
dask
client from <http://github.com/bstadlbauer/dask-k8s-operator-go-client|github.com/bstadlbauer/dask-k8s-operator-go-client>
with the official one from <http://github.com/dask/dask-kubernetes/v2023|github.com/dask/dask-kubernetes/v2023>
.
The operator has been moved over from my repo to the official one in dask/dask-kubernetes#731, so the code is essentially the same, but I'm going ahead and will archive my version.
Type
☐ Bug Fix
☐ Feature
☑︎ Plugin
flyteorg/flyteplugins
Codecov: 62.75% (-1.31%) compared to dfdf6f9
✅ 6 other checks have passed
6/7 successful checksGitHub
06/27/2023, 10:37 AMimport os
import pathlib
from flytekitplugins.papermill import NotebookTask
from flytekitplugins.awsbatch import AWSBatchConfig
config = AWSBatchConfig(
platformCapabilities="EC2",
)
backtest_wages = NotebookTask(
name="test-notebook",
notebook_path=os.path.join(
pathlib.Path(__file__).parent.parent.absolute(), "notebooks/test-notebook.ipynb"
),
render_deck=True,
task_config=config
)
Tried to use AWS batch job to run the notebook task using the code example above, got an error:
logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'
When I looked at the batch job docker command:
["pyflyte-execute","--inputs","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/inputs.pb>","--output-prefix","<s3://fiolito-eks-cluster/metadata/propeller/liquidity-forecast-development-alg58fr86fmkgh9nqnf4/n0/data/0/0>","--raw-output-data-prefix","<s3://fiolito-eks-cluster/99/alg58fr86fmkgh9nqnf4-n0-0>","--resolver","flytekit.core.python_auto_container.default_task_resolver","--","task-module","flytekitplugins.papermill.task","task-name","_dummy_task_func"]
It seems that the _dummy_task_func is wrong which should be test-notebook
and it comes from:
https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py#L145
Expected behavior
Notebook task can run as an AWS Map Task without throwing exception.
Additional context to reproduce
Python3.10
flytekit 1.7.0
flytekitplugin-awsbatch 1.7.0
flytekitplugin-papermil 1.7.0
Screenshots
image▾
image▾
image▾
GitHub
06/27/2023, 3:26 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/5a5ad821b7ba3c51166a40c11d1695fb79a7ea8d|5a5ad821>
- Fix map task cache misses (#363)
flyteorg/flytepluginsGitHub
06/27/2023, 3:27 PMGitHub
06/27/2023, 3:29 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/2c393c030f44a87a4b67a859ffc6059fd8de50fe|2c393c03>
- Switch to official dask operator (#366)
flyteorg/flytepluginsGitHub
06/27/2023, 3:29 PMGitHub
06/27/2023, 5:33 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/1052af8c1a4eff426023db19e1fba4441a03e490|1052af8c>
- Improve task type hint (#1711)
flyteorg/flytekitGitHub
06/27/2023, 5:58 PM<https://github.com/flyteorg/flyteidl/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteidl/commit/b219c2ab37886801039fda67d913760ac6fc4c8b|b219c2ab>
- Add ArrayNode proto message (#390)
flyteorg/flyteidlGitHub
06/27/2023, 6:02 PM