GitHub
03/07/2023, 9:18 PMGitHub
03/07/2023, 9:21 PMGitHub
03/07/2023, 9:27 PMrelease-*
.
flyteorg/flytekit
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (windows-latest, 3.7)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.7)
DCO: DCO
✅ 1 other check has passed
1/9 successful checksGitHub
03/07/2023, 10:16 PMGitHub
03/07/2023, 10:22 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/f8d538931f0baf3da453d3fa9c7ec22e33a7c937|f8d53893>
- docs: add empty file to shell_task.py flytedirectory (#964)
flyteorg/flytesnacksGitHub
03/07/2023, 10:43 PM<https://github.com/flyteorg/flytectl/tree/master|master>
by pmahindrakar-oss
<https://github.com/flyteorg/flytectl/commit/6f376a3cf7745468a24bb1f3397fa5d759f3df9a|6f376a3c>
- Doc updates (#398)
flyteorg/flytectlGitHub
03/07/2023, 10:52 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/be24c52f3fb8f7949db172f490cca95fa0f0e413|be24c52f>
- Pin fsspec to <=2023.1 (#1537)
flyteorg/flytekitGitHub
03/07/2023, 11:09 PMGitHub
03/07/2023, 11:19 PMGitHub
03/08/2023, 1:21 AMGitHub
03/08/2023, 1:43 AMGitHub
03/08/2023, 2:55 AMGitHub
03/08/2023, 3:06 AMGitHub
03/08/2023, 5:03 AMGitHub
03/08/2023, 7:49 AM$ docker run -it --platform linux/arm64 -u root <http://cr.flyte.org/flyteorg/flyteadmin-release:latest|cr.flyte.org/flyteorg/flyteadmin-release:latest> sh
/ # apk add file
/ # file bin/flyteadmin
bin/flyteadmin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=pwwxVfCtJjTsZm9XA0ED/BJdCEuBm0QjhMujOZ7E7/a_vHl8mBZ4xrHPFYqAdf/on6kzeFZfsYdRjhPjzhK, stripped
Other images also have this issue, for example:
$ docker run -it --platform linux/arm64 -u root <http://cr.flyte.org/flyteorg/datacatalog:latest|cr.flyte.org/flyteorg/datacatalog:latest> sh
/ # apk add file
/ # file bin/datacatalog
bin/datacatalog: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=YBzDJD8nguzh9dRW4GPp/iKNJqrMQzbqKL7lVR_L0/_wLIwVTI2EO3sXJO2Sup/5m-2xGfqxh4rTZlXehX4, not stripped
I did not check every images, but I'm pretty sure that most images have the issue.
Expected behavior
ARM64 Docker Images should not contain x86_64 binaries. They should contain arm64 binaries instead. Graviton instances should be able to run flyte using docker.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/08/2023, 8:01 AMAnnotated[FlyteFile, HashMethod(...)]
for calculating a custom hash for a FlyteFile
doesn't work out-of-the-box, failing an `issubclass` check in flytekit/types/file/file.py.
This makes it impossible to use custom hash functions with FlyteFile
, e.g., to implement content-aware caching.
Note, that the same approach with Annotated[FlyteDirectory, HashMethod(...)]
works as expected out of the box.
Example workflow code
import hashlib
from typing import Annotated
import pandas as pd
from flytekit import task, workflow
from flytekit.core.hash import HashMethod
from flytekit.types.file import FlyteFile
def calc_hash(f: FlyteFile) -> str:
"""Calculate SHA1 hash of a file"""
h = hashlib.sha1(usedforsecurity=False)
with open(f.path, "rb") as f:
while chunk := f.read(4096):
h.update(chunk)
return str(h.hexdigest())
CachedFlyteFile = Annotated[FlyteFile, HashMethod(calc_hash)]
@task
def write_file() -> CachedFlyteFile:
print("write_file")
local_path = "data.parquet"
df = pd.DataFrame(data={"a": [1, 2, 3], "b": [3, 4, 5]})
df.to_parquet(local_path)
return FlyteFile(local_path, remote_path=f"<s3://test-repo/main/{local_path}>")
@task(cache=True, cache_version="1")
def print_file(file: FlyteFile) -> None:
file.download()
print(pd.read_parquet(file))
@workflow
def wf() -> None:
f = write_file()
print_file(file=f)
if __name__ == "__main__":
wf()
wf() # don't expect output from `print_file`, since it should be cached
Expected behavior
The first execution of wf
should run both the write_file
and print_file
tasks, the second execution should only run write_file
and hit the cache for print_file
.
In reality, an exception is raised in `flytekit/types/file/file.py, in to_literal()`:
Additional context to reproduce See example workflow above. I was able to solve the issue by inserting the following before theTypeError: issubclass() arg 1 must be a class
issubclass
check in flytekit/types/file/file.py
(I haven't run the flytekit test suite, so this might not be suitable for a mergeable PR just yet - should I submit a draft PR regardless?):
@@ -284,6 +288,10 @@
"None value cannot be converted to a file."
)
+ # Handle Annotated[FlyteFile, ...] correctly by extracting the wrapped type
+ if issubclass(typing.get_origin(python_type), typing.Annotated):
+ python_type = typing.get_args(python_type)[0]
+
if not (python_type is os.PathLike or issubclass(python_type, FlyteFile)):
raise ValueError(
f"Incorrect type {python_type}, must be either a FlyteFile or os.PathLike"
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/08/2023, 9:32 AM<https://github.com/flyteorg/flytectl/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytectl/commit/e55c64995f4ebce251130a730945abc3f3430d07|e55c6499>
- Fix docs for updating description using workflow-meta subcommand (#394)
flyteorg/flytectlGitHub
03/08/2023, 9:47 AMGitHub
03/08/2023, 9:55 AMGitHub
03/08/2023, 9:55 AM<https://github.com/flyteorg/homebrew-tap/tree/main|main>
by flyte-bot
<https://github.com/flyteorg/homebrew-tap/commit/bcf44fbdf1f23b1db4f79711e8a6232710a107a3|bcf44fbd>
- Brew formula update for flytectl version v0.6.33
flyteorg/homebrew-tapGitHub
03/08/2023, 2:10 PMSUCCEEDED
phase, but on recovery this will be in the RECOVERED
phase.
recovered▾
GitHub
03/08/2023, 2:11 PMGitHub
03/08/2023, 3:09 PM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytesnacks/commit/641195c097bfb6ee6a7778d4f5dc9d4643d903e5|641195c0>
- DuckDB integration docs (#966)
flyteorg/flytesnacksGitHub
03/08/2023, 4:01 PMGitHub
03/08/2023, 5:51 PM<https://github.com/flyteorg/flyte/tree/master|master>
by jeevb
<https://github.com/flyteorg/flyte/commit/6b04701b3fe8f1b39d63952a99863c11d865c83b|6b04701b>
- Enable workflow dispatch for single-binary image builds (#3422)
flyteorg/flyteGitHub
03/08/2023, 6:12 PMGitHub
03/08/2023, 6:28 PM--version
flag) that makes sure that it's URL-safe. We want to validate that the version id consists of only:
• alphanumeric characters
• .
, -
, _
, %
Otherwise, version ids with, for example /
, won't be rendered correctly on the Flyte UI.
Expected behavior
If a user tries to do something like
pyflyte register ... --version my/version/1
• flytekit should raise an error.
• flytectl register
should also raise an error
_Edit_: the validation should occur in admin, so that we don't do repeat validation across clients.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/08/2023, 6:35 PMGitHub
03/08/2023, 6:36 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/3bc3714ce653c634a0dddcde646f3a84f17d2d2e|3bc3714c>
- fix: fix sed makefile error (#714)
flyteorg/flyteconsoleGitHub
03/08/2023, 6:56 PM