GitHub
01/24/2023, 2:42 PMconf.py
(dask.py
) so that it matches the actual name of the file (dask_example.py
) in integrations/kubernetes/k8s_dask/
enabling the page to appear in the left nav.
flyteorg/flytesnacks
✅ All checks have passed
2/2 successful checksGitHub
01/24/2023, 2:53 PMSdkTransform
using the withInput(...)
method that forces you to know the attribute names of the input class.
Example
previous way
SdkNode<SumTask.SumOutput> sumNode =
builder.apply("fib-3", new SumTask().withInput("fib1", fib1).withInput("fib2", fib2));
new way
SdkNode<SumTask.SumOutput> sumNode =
builder.apply("fib-3", new SumTask(), SumTask.SumInput.create(fib1, fib2));
Describe alternatives you've considered
None
Propose: Link/Inline OR Additional context
flyteorg/flytekit-java#175
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/24/2023, 2:56 PMGitHub
01/24/2023, 4:49 PMremote.execute
stopped working with exception:
...
# We need to manually make a map of the nodes since there is none for single task executions
# Assume the first one is the only one.
node_mapping = (
{
> task_node_exec[0].id.node_id: FlyteNode(
id=flyte_entity.id,
upstream_nodes=[],
bindings=[],
metadata=NodeMetadata(name=""),
flyte_task=flyte_entity,
)
}
if len(task_node_exec) >= 1
else {} # This is for the case where node executions haven't appeared yet
)
E TypeError: __init__() got an unexpected keyword argument 'flyte_task'
.tox/py38-test/lib/python3.8/site-packages/flytekit/remote/remote.py:1454: TypeError
As commented in flyteorg/flytekit#1348 (comment), it seems to be that PR breaking it during refactoring.
Expected behavior
remote.execute
should work
Additional context to reproduce
• Upgrade to 1.3.0
• Do the following
execution = remote.execute(
...
)
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/24/2023, 6:11 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/99722d59349a574a73e4b77e77bfa54ceab5c012|99722d59>
- Return error code on fail (#1408)
flyteorg/flytekitGitHub
01/24/2023, 6:57 PMGitHub
01/24/2023, 7:59 PMk8s-array
plugin exposes it's own logging configuration. This requires users to define log configuration in multiple places.
Goal: What should the final outcome look like, ideally?
Map tasks should default to using the globally defined configuration and only use the k8s-array
log configuration as an override.
Describe alternatives you've considered
Leaving it as is results in unintended behavior.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/24/2023, 9:04 PMGitHub
01/24/2023, 9:20 PMflytekit.extras.sklearn
in flytekit/__init__.py
so that the sklearn type transformer is registered automatically if scikit-learn is installed.
• Also fix up the sklearn/pytorch/tensorflow docs
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytekit
Codecov: 69.16% (-0.09%) compared to 7b48fff
✅ 29 other checks have passed
29/30 successful checksGitHub
01/24/2023, 9:24 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/9609dad0991ca0637a833ab3ee8eb25af876f03a|9609dad0>
- Bump certifi from 2022.9.24 to 2022.12.7 (#3143)
flyteorg/flyteGitHub
01/24/2023, 9:24 PM@dependabot rebase
.
* * *
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
• @dependabot rebase
will rebase this PR
• @dependabot recreate
will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge
will merge this PR after your CI passes on it
• @dependabot squash and merge
will squash and merge this PR after your CI passes on it
• @dependabot cancel merge
will cancel a previously requested merge and block automerging
• @dependabot reopen
will reopen this PR if it is closed
• @dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels
will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers
will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees
will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone
will set the current milestone as the default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the Security Alerts page.
flyteorg/flyte
✅ All checks have passed
6/6 successful checksGitHub
01/24/2023, 9:43 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by kumare3
<https://github.com/flyteorg/flytekit/commit/5126b2aa86c3b2b3154eeb782e71c97cde95ac9a|5126b2aa>
- wrapping flyte entity in a task node in call to flyte node constructor, not sure if integration tests are actually running (#1422)
flyteorg/flytekitGitHub
01/25/2023, 12:19 AMGitHub
01/25/2023, 2:16 AMmake setup
as part of the dev environment, however that command fails on Mac M1's with this error message:
ERROR: Ignored the following versions that require a different python version: 2.6.2 Requires-Python >=3.6, <3.9
ERROR: Could not find a version that satisfies the requirement tensorflow-io-gcs-filesystem==0.28.0 (from versions: none)
ERROR: No matching distribution found for tensorflow-io-gcs-filesystem==0.28.0
and
ERROR: No matching distribution found for tensorflow==2.8.1
The issue stems from the fact that tensorflow special-cases a dependency based on the platform+OS, while we currently only maintain a single set of requirements files.
Expected behavior
Mac M1's users should be able to run make setup
successfully.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/25/2023, 2:25 AMmake dev-requirements.txt
on a M1 and push a commit to this PR with the contents of the dev-requirements-mac_arm64.txt
file.
☐ run make setup
on a new venv
Tracking Issue
flyteorg/flyte#3264
Follow-up issue
NA
flyteorg/flytekit
WIP: Title contains "WIP"
✅ 29 other checks have passed
29/30 successful checksGitHub
01/25/2023, 4:31 AMv0.23.7
What's Changed
• C API stable test by `@gjtorikian` in gjtorikian/commonmarker#201
• Update to 29.0.gfm.7 by `@anticomputer` in gjtorikian/commonmarker#224
Full Changelog: gjtorikian/commonmarker@v0.23.6...v0.23.7
v0.23.7.pre1
What's Changed
• C API stable test by `@gjtorikian` in gjtorikian/commonmarker#201
Full Changelog: gjtorikian/commonmarker@v0.23.6...v0.23.7.pre1Changelog Sourced from commonmarker's changelog.
Changelog
v1.0.0.pre6 (2023-01-09)
Full Changelog
Closed issues:
• Cargo.lock prevents Ruby 3.2.0 from installing commonmarker v1.0.0.pre4 #211
Merged pull requests:
• always use rb_sys (don't use Ruby's emerging cargo tooling where available) #213 (kivikakk)
v1.0.0.pre5 (2023-01-08)
Full Changelog
Merged pull requests:
• Provide 3.2 build support #212 (gjtorikian)
v1.0.0.pre4 (2022-12-28)
Full Changelog
Closed issues:
• Will the cmark-gfm branch continue to be maintained for awhile? #207
Merged pull requests:
• Implement native syntax highlighting #209 (gjtorikian)
• Bump magnus from 0.4.3 to 0.4.4 #208 (dependabot[bot])
• Bump magnus from 0.4.2 to 0.4.3 #206 (dependabot[bot])
• Bump comrak from 0.14.0 to 0.15.0 #205 (dependabot[bot])
• Bump magnus from 0.4.1 to 0.4.2 #204 (dependabot[bot])
v1.0.0.pre3 (2022-11-30)
Full Changelog
Closed issues:
• Code block incorrectly parsed in commonmarker 1.0.0.pre #202
Merged pull requests:
• Windows build #197 (gjtorikian)... (truncated) Commits • `734fd86` Merge pull request #224 from gjtorikian/update-to-29.0.gfm.7 • `2e724ec` Turned off Rubocop. • `9c923b0` 💎 release 0.23.7 • `30419c2` Added call to cmark_init_standard_node_flags() • `9007c37` Update cmark-upstream to github/cmark-gfm@57d5e093e... • `1cfec13` Merge pull request #201 from gjtorikian/c-api-stable-test • `bbf631b` lint • `5b807a1` ease up • `9a24e6d` Test fake version • `d8a43bc` Allow for manual dispatch • Additional commits viewable in compare view Dependabot compatibility score Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.
* * *
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
• @dependabot rebase
will rebase this PR
• @dependabot recreate
will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge
will merge this PR after your CI passes on it
• @dependabot squash and merge
will squash and merge this PR after your CI passes on it
• @dependabot cancel merge
will cancel a previously requested merge and block automerging
• @dependabot reopen
will reopen this PR if it is closed
• @dependabot close
will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version
will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version
will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency
will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels
will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers
will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees
will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone
will set the current milestone as the default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the <https:/…
flyteorg/flyteorg.github.io
✅ All checks have passed
2/2 successful checksGitHub
01/25/2023, 5:51 AMfrom flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
r = FlyteRemote(
Config.auto(config_file="/Users/ytong/.flyte/dev.yaml"),
default_project="flytesnacks",
default_domain="development",
)
execution_id = "asfv42v66gsxm5hljfg8"
ex = r.fetch_execution(name=execution_id)
Sync'ing nodes returns this error
In [5]: r.sync_execution(ex, sync_nodes=True)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-cc8f74e15715> in <cell line: 1>()
----> 1 r.sync_execution(ex, sync_nodes=True)
~/go/src/github.com/flyteorg/flytekit/flytekit/remote/remote.py in sync_execution(self, execution, entity_definition, sync_nodes)
1478 node_execs = {}
1479 for n in underlying_node_executions:
-> 1480 node_execs[n.id.node_id] = self.sync_node_execution(n, node_mapping) # noqa
1481 execution._node_executions = node_execs
1482 return self._assign_inputs_and_outputs(execution, execution_data, node_interface)
~/go/src/github.com/flyteorg/flytekit/flytekit/remote/remote.py in sync_node_execution(self, execution, node_mapping)
1620 for t in iterate_task_executions(self.client, execution.id)
1621 ]
-> 1622 execution._interface = execution._node.flyte_entity.interface
1623
1624 self._assign_inputs_and_outputs(
AttributeError: 'FlyteGateNode' object has no attribute 'interface'
Expected behavior
should not error.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/25/2023, 6:18 AM/var/lib/flyte
where data with persistence requirements might be stored. HostPath PV/PVCs are also created for PostgreSQL and Minio at /var/lib/flyte/storage/db
and /var/lib/flyte/storage/minio
respectively. Mounting a pre-created (via docker volume create flyte-sandbox
) docker volume to /var/lib/flyte
will ensure that data will persist across different cluster versions, for as long as the docker volume exists.
TODO: Logic will need to be added to flytectl
to manage the lifecycle of the docker volume, and ensure that it is correctly mounted into the sandbox cluster.
flyteorg/flyte
✅ All checks have passed
11/11 successful checksGitHub
01/25/2023, 6:54 AM{"json":{},"level":"error","msg":"Container [<gcp-bucket-name>] lookup failed. Error Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`%22,%22ts%22:%222023-01-23T12:56:22Z%22}|https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","ts":"2023-01-23T12:56:22Z"}>
{"json":{"app_name":"datacatalog"},"level":"error","msg":"Failed to create DataStore \u0026{stow {{{ false }} iam us-east-1 false} {google map[json: project_id:prj-moj-p-ds-training scopes:<https://www.googleapis.com/auth/devstorage.read_write>]} <gcp-bucket-name> false {0 0} {10} {map[] 0s} {map[]}}, err Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>e?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`%22,%22ts%22:%222023-01-23T12:56:22Z%22}|https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","ts":"2023-01-23T12:56:22Z"}>
{"json":{},"level":"fatal","msg":"caught panic: Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n>` [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:86|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:86> +0x9c\npanic({0x1222fc0, 0xc000a148a0})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:94|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:94> +0x906\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00093fca8|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00093fca8>?, 0x474ddc?}, 0xc000143fb0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:143 +0x33\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x16bba38|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x16bba38>, 0xc000046018}, 0xc000143fb0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:129 +0x45\<http://ngithub.com/flyteorg/datacatalog/cmd/entrypoints.glob..func3(0x1f7f6e0|ngithub.com/flyteorg/datacatalog/cmd/entrypoints.glob..func3(0x1f7f6e0>?, {0x1382675?, 0x2?, 0x2?})\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/serve.go:33 +0x115\<http://ngithub.com/spf13/cobra.(*Command).execute(0x1f7f6e0|ngithub.com/spf13/cobra.(*Command).execute(0x1f7f6e0>, {0xc0002b4540, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1f7f460)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974|ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1f7f460)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974> +0x3b4\<http://ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\ngithub.com/flyteorg/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46|ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\ngithub.com/flyteorg/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46> +0x25\nmain.main()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/main.go:10 +0x6b\n]","ts":"2023-01-23T12:56:22Z"}
Expected behavior
The deployment should be up and running as other deployments with similar requirement are up with same account.
I ensured that datacatalog Serviceaccount has binding to corresponding GSA similar to other deployments.
Observation: I see for other deployments corresponding cluster-role and cluster-role-bindings are created, but its missing for datacatalog, isn't it required for datacatalog as well ?
flyte-flyte-pod-webhook flyte-pod-webhook
flyte-flyteadmin-binding flyteadmin
flyte-flytepropeller flytepropeller
Additional context to reproduce
Following GCP manual
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
01/25/2023, 9:33 AMGitHub
01/25/2023, 10:16 AMGitHub
01/25/2023, 10:33 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by andresgomezfrr
<https://github.com/flyteorg/flytekit-java/commit/2a40a16d9d21d686891ec8895bd05829e4b1be0c|2a40a16d>
- Add typed transform inputs (#175)
flyteorg/flytekit-javaGitHub
01/25/2023, 10:37 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/f4b71b4c9730939ed3709d0c954b8dfd53a96571|f4b71b4c>
- [maven-release-plugin] prepare release 0.4.0
flyteorg/flytekit-javaGitHub
01/25/2023, 10:37 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by github-actions[bot]
<https://github.com/flyteorg/flytekit-java/commit/fb64805b1e087d475adcd92ec18c509264b7ec63|fb64805b>
- [maven-release-plugin] prepare for next development iteration
flyteorg/flytekit-javaGitHub
01/25/2023, 11:11 AMGitHub
01/25/2023, 11:17 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by sonjaer
<https://github.com/flyteorg/flytekit-java/commit/b45689e07d3f5ab4a6e31474b3e611770aaee61a|b45689e0>
- Add remote example (#178)
flyteorg/flytekit-javaGitHub
01/25/2023, 11:24 AMGitHub
01/25/2023, 11:57 AM<https://github.com/flyteorg/flytekit-java/tree/master|master>
by sonjaer
<https://github.com/flyteorg/flytekit-java/commit/f99b7d302999ca4c32d49773252a2acf9b9c23bf|f99b7d30>
- Add remote example for launch plan (#179)
flyteorg/flytekit-javaGitHub
01/25/2023, 1:27 PMPods
marked as a retryable failure are never cleaned up. This PR only fast-fails when aborting tasks in a success phase, to ensure that all resources are correctly cleaned up at the risk of unnecessary API calls (ex. object does not exist, etc).
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
Currently, the TaskHandler
fast-fails when aborting resources in a terminal phase. This causes issues where resources fail to cleanup when the task is reported as failed (ie. retryable failure or permanent failure) but the resource remains running.
One such scenario is handling Pods
in the "ImagePullBackoff" state. We mark these as failed, but because of the fast-fail during abort these k8s resources are never deleted.
This may occur within other plugins as well. To be safe, we may want to attempt to abort all task phases other than Success
.
Tracking Issue
fixes flyteorg/flyte#3239
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
01/25/2023, 4:35 PM<https://github.com/flyteorg/datacatalog/tree/master|master>
by hamersaw
<https://github.com/flyteorg/datacatalog/commit/24d8aba55130a83bb83b845d98aed7399aa651b6|24d8aba5>
- enabling connection pooling (#89)
flyteorg/datacatalog