GitHub
04/12/2023, 5:43 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by jeevb
<https://github.com/flyteorg/flytepropeller/commit/b9d1f621f2de0ba20d672561251f4900c26a1847|b9d1f621>
- fix makefile to read variables from environment and overrides (#554)
flyteorg/flytepropellerGitHub
04/12/2023, 5:43 PM<https://github.com/flyteorg/datacatalog/tree/master|master>
by jeevb
<https://github.com/flyteorg/datacatalog/commit/0638a9e764be7ba11792985d7d09bed4ef5da375|0638a9e7>
- fix makefile to read variables from environment and overrides (#104)
flyteorg/datacatalogGitHub
04/12/2023, 5:44 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyteadmin/commit/97a6aa4e8c91a570ae480a2060402d54508e1ae1|97a6aa4e>
- Infer GOOS and GOARCH from environment (#550)
flyteorg/flyteadminGitHub
04/12/2023, 5:49 PM<https://github.com/flyteorg/flyte/tree/master|master>
by wild-endeavor
<https://github.com/flyteorg/flyte/commit/0bd4ab919b2ac49dbad888d0fe4195d02af5bf24|0bd4ab91>
- Fix: Stow scope for signing URLs in GKE >= 1.25.0 (#3383)
flyteorg/flyteGitHub
04/12/2023, 5:49 PM<https://www.googleapis.com/auth/devstorage.read_write>
seems to be no longer enough in GKE versions >=1.25.0. We could not find any release notes from Google pointing in this direction but noticed that Flyte would no longer work after we updated the GKE cluster version to 1.25.5.
Fast registration failed with this error response from `flyteadmin`:
..
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 580, in fast_package
return self._upload_file(pathlib.Path(zip_file))
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/remote/remote.py", line 598, in _upload_file
upload_location = self.client.get_upload_signed_url(
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 41, in handler
return fn(*args, **kwargs)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/flytekit/clients/raw.py", line 856, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/opt/pyenv-root/versions/3.9.12/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.
Details:
[
{
"@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
"domain": "<http://googleapis.com|googleapis.com>",
"metadata": {
"method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
"service": "<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>"
},
"reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
}
]"
debug_error_string = "UNKNOWN:Error received from peer ipv4:{removed} {created_time:"2023-02-20T11:01:31.243469439+00:00", grpc_status:13, grpc_message:"failed to create a signed url. Error: unable to sign bytes: googleapi: Error 403: Request had insufficient authentication scopes.\nDetails:\n[\n {\n \"@type\": \"<http://type.googleapis.com/google.rpc.ErrorInfo\|type.googleapis.com/google.rpc.ErrorInfo\>",\n \"domain\": \"<http://googleapis.com|googleapis.com>\",\n \"metadata\": {\n \"method\": \"google.iam.credentials.v1.IAMCredentials.SignBlob\",\n \"service\": \"<http://iamcredentials.googleapis.com|iamcredentials.googleapis.com>\"\n },\n \"reason\": \"ACCESS_TOKEN_SCOPE_INSUFFICIENT\"\n }\n]"}"
Check all the applicable boxes
☐ I updated the documentation accordingly. (no documentation needs to be updated?)
☑︎ All new and existing tests passed. (I ran the helm and kustomize test targets)
☑︎ All commits are signed-off.
Note to reviewers
We started a discussion in Slack on this issue. @Yee .
As far as we know there is no more restrictive scope available. We checked this Google page and it seems like google.iam.credentials.v1.IAMCredentials.SignBlob
is only available in the cloud-platform
scope.
We also tried <https://www.googleapis.com/auth/devstorage.full_control>
and it was not enough.
flyteorg/flyte
GitHub Actions: trigger-sandbox-bundled-build
GitHub Actions: trigger-single-binary-build
✅ 10 other checks have passed
10/12 successful checksGitHub
04/12/2023, 5:55 PMGitHub
04/12/2023, 5:56 PM<https://github.com/flyteorg/flytecopilot/tree/master|master>
by jeevb
<https://github.com/flyteorg/flytecopilot/commit/a913fde9516d649085ba7b977fbc46e3790b0ea8|a913fde9>
- Enable multiarch build and infer GOOS and GOARCH from environment (#57)
flyteorg/flytecopilotGitHub
04/12/2023, 6:32 PMGitHub
04/12/2023, 6:33 PMGitHub
04/12/2023, 6:35 PMGitHub
04/12/2023, 8:14 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/530c10dacce9b1e38c01487da48f03488c02a618|530c10da>
- fix: passthrough runtime env vars (#741)
flyteorg/flyteconsoleGitHub
04/12/2023, 8:35 PMpyflyte register workflows --project repo --image localhost:30000/$FLYTE_DOCKER_IMAGE_TAG
Running pyflyte register from ~/Dev/repo with images ImageConfig(default_image=Image(name='default', fqn='localhost:30000/repo', tag='imageTag'), images=[Image(name='default', fqn='localhost:30000/repo', tag='imageTag')]) and image destination folder /root on 1 package(s) ('~/Dev/repo/workflows',)
Registering against localhost:30080
Detected Root ~/Dev/repo, using this to create deployable package...
No output path provided, using a temporary directory at /var/folders/t4/7xnpxvjj3w3g7cf752g1qvjh0000gn/T/tmp7dmep_68 instead
Computed version is kChHd1ibpkx7jpAR1MIWIg==
Loading packages ['workflows'] under source root ~/Dev/repo
Successfully serialized 0 flyte objects
No Flyte entities were detected. Aborting!
Expected behavior
I would expect this to work the way the python example (downloadable from that same page in the docs) works:
pyflyte register workflows --project repo --image localhost:30000/$FLYTE_DOCKER_IMAGE_TAG
Running pyflyte register from ~/Dev/repo with images ImageConfig(default_image=Image(name='default', fqn='localhost:30000/repo', tag='imageTag'), images=[Image(name='default', fqn='localhost:30000/repo', tag='imageTag')]) and image destination folder /root on 1 package(s) ('~/Dev/repo/workflows',)
Registering against localhost:30080
Detected Root ~/Dev/repo, using this to create deployable package...
No output path provided, using a temporary directory at /var/folders/t4/7xnpxvjj3w3g7cf752g1qvjh0000gn/T/tmpq6mm2c8o instead
Computed version is 69UqeNHQfi0GyndHMSA6uw==
Loading packages ['workflows'] under source root /Users/peterklingelhofer/Dev/am-etl-poc
Successfully serialized 3 flyte objects
[✔] Registration workflows.ray_example.ray_task type TASK successful with version 69UqeNHQfi0GyndHMSA6uw==
[✔] Registration workflows.ray_example.ray_workflow type WORKFLOW successful with version 69UqeNHQfi0GyndHMSA6uw==
[✔] Registration workflows.ray_example.ray_workflow type LAUNCH_PLAN successful with version 69UqeNHQfi0GyndHMSA6uw==
Successfully registered 3 entities
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/12/2023, 8:35 PM<https://github.com/flyteorg/flyte/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flyte/commit/c66d37ca9fce42ffb4a0c1550583665873862b0a|c66d37ca>
- fixes copy button issue (#3584)
flyteorg/flyteGitHub
04/12/2023, 8:44 PMGitHub
04/12/2023, 9:23 PM<https://github.com/flyteorg/flytekit-python-template/tree/main|main>
by zeryx
<https://github.com/flyteorg/flytekit-python-template/commit/49612b39468aad48f084f8e25fd5549ec483ce1f|49612b39>
- added github hash as secondary docker push for each image (#28)
flyteorg/flytekit-python-templateGitHub
04/12/2023, 9:39 PMgo.mod
, so we will only have one go.mod
in root directory
• Single image for all the component.
• Single binary to start different component. For example, flyte start admin
, flyte start propeller
.
• Move single binary to /flyte-single-binary
folder.
• Update generate_flyte_manifest.yml
flyteorg/flyte
GitHub Actions: build-and-push-single-binary-image
GitHub Actions: unit-tests (flytestdlib)
GitHub Actions: unit-tests (flyteplugins)
GitHub Actions: unit-tests (flytecopilot)
GitHub Actions: unit-tests (datacatalog)
GitHub Actions: unit-tests (flyteadmin)
GitHub Actions: unit-tests (flytepropeller)
GitHub Actions: Push to Github Registry
GitHub Actions: Push to Github Registry
GitHub Actions: lint (flytestdlib)
GitHub Actions: lint (flyteplugins)
GitHub Actions: unit-tests (flytestdlib)
GitHub Actions: lint (flytecopilot)
GitHub Actions: unit-tests (flyteplugins)
GitHub Actions: lint (datacatalog)
GitHub Actions: unit-tests (flytecopilot)
GitHub Actions: lint (flyteadmin)
GitHub Actions: lint (flytepropeller)
GitHub Actions: unit-tests (flyteadmin)
GitHub Actions: integration-tests
GitHub Actions: unit-tests (flytepropeller)
GitHub Actions: trigger-sandbox-lite-build
GitHub Actions: go-generate (flytestdlib)
GitHub Actions: go-generate (flyteplugins)
GitHub Actions: go-generate (flytecopilot)
GitHub Actions: go-generate (datacatalog)
GitHub Actions: go-generate (flyteadmin)
✅ 3 other checks have passed
3/30 successful checksGitHub
04/12/2023, 9:56 PM<https://github.com/flyteorg/flytekit-python-template/tree/main|main>
by zeryx
<https://github.com/flyteorg/flytekit-python-template/commit/7129df62290bc13375ba29552391133d67d39ad4|7129df62>
- missed the -latest flag
flyteorg/flytekit-python-templateGitHub
04/12/2023, 9:58 PMGitHub
04/13/2023, 3:45 PM@task
...
PodTemplate(
pod_spec = V1PodSpec(
containers=[],
tolerations=[
V1Toleration(<some toleration>)
],
node_selector=<some nodeselector>
)
)
... such a Pod is created:
apiVersion: v1
kind: Pod
...
spec:
containers:
- image: ...
name: primary
nodeSelector:
<some nodeselector>
tolerations:
- <some toleration>
The nodeselector and toleration is applied correctly but the primary container name is primary
instead of the typical task id.
For a normal task running in a single Pod, the container name doesn't really matter. However, this leads to problems e.g. with PytorchJobs for which the primary container name is expected to be pytorch
.
Here, a typical task id-based container name would be replaced with pytorch
, the string primary
is not.
Expected behavior
Here, the container name in target.K8SPod.PodSpec
should not be primary
.
If this were the case, it would correctly be replaced with pytorch
here.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/13/2023, 4:17 PMRUNNING
Screenshot 2023-04-13 at 9 11 06 AM▾
Screenshot 2023-04-13 at 9 11 12 AM▾
GitHub
04/13/2023, 4:21 PM<https://github.com/flyteorg/flytectl/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytectl/commit/bd0070609ebd1f14f1498412c39a3d1f14125cac|bd007060>
- Update workflow_execution_config (#401)
flyteorg/flytectlGitHub
04/13/2023, 5:04 PMGitHub
04/13/2023, 5:04 PM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/267728127888aa16b1bee501533baca7ca720965|26772812>
- Brew formula update for flytectl version v0.6.35
flyteorg/homebrew-tapGitHub
04/14/2023, 2:01 AMTerminateExecution
error when a workflow is already in terminal state is PermissionDenied
. This changes it to return PreconditionFailed
w/ specific grpc status message.
Type
☑︎ Bug Fix
☐ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Tracking Issue
fixes flyteorg/flyte#3582
flyteorg/flyteadmin
GitHub Actions: Build & Push Flytescheduler Image
GitHub Actions: Build & Push Flyteadmin Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
Codecov: 58.85% (-1.24%) compared to a89b359
✅ 9 other checks have passed
9/14 successful checksGitHub
04/14/2023, 2:03 AMGitHub
04/14/2023, 4:28 AMdisable_deck parameter=False
. For example:
import flytekit
from flytekit import task, Resources, workflow
from flytekit.core.utils import MeasureExecutionTime
@task(
disable_deck=False,
limits=Resources(mem="4Gi",cpu="1"),
)
def t1():
import time
# Test MeasureExecutionTime used as a decorator
@MeasureExecutionTime("download data from s3")
def download_data():
time.sleep(1)
download_data()
# Test MeasureExecutionTime used as a context manager
# Simulate a user using a very long name for a time measurement
with MeasureExecutionTime("test long string"*10):
time.sleep(1)
# Simulate multiple time measurements
for i in range(10):
with MeasureExecutionTime(f"small tasks {i}"):
# The 'print' function will execute quickly compared to 'time.sleep(1)'.
# This is used to test if the timeline graph can accurately represent
# measurements with significantly varying execution times.
print("hello world")
@workflow
def wf():
t1()
if __name__ == "__main__":
wf()
Test in sandbox:
image▾
image▾
image▾
image▾
GitHub
04/14/2023, 2:25 PMGitHub
04/14/2023, 2:28 PMGitHub
04/14/2023, 2:31 PMGitHub
04/14/2023, 3:02 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flytepropeller/commit/0c982eaa17301bddd7c27ef0e1c6165a0435431c|0c982eaa>
- Remove BarrierTick (#545)
flyteorg/flytepropeller