GitHub
04/14/2023, 3:04 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/435436bd112bf7869c83bd5e951ab2d9cf5166ca|435436bd>
- Set PrimaryContainerKey annotation by default (#337)
flyteorg/flytepluginsGitHub
04/14/2023, 3:22 PMGitHub
04/14/2023, 3:51 PMPod
operator to detect container completion and mark the Pod
as completed.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☐ Unit tests added
☑︎ Code documentation added
☐ Any pending items have an associated Issue
Complete description
^^^
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Goreleaser
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
04/14/2023, 4:02 PMGitHub
04/14/2023, 4:13 PMValueError: Empty module name
args:[24 items
0:"pyflyte-fast-execute"
1:"--additional-distribution"
2:"s3://...."
3:"--dest-dir"
4:"{{ .dest_dir }}"
5:"--"
6:"pyflyte-execute"
7:"--inputs"
8:"{{.input}}"
9:"--output-prefix"
10:"{{.outputPrefix}}"
11:"--raw-output-data-prefix"
12:"{{.rawOutputDataPrefix}}"
13:"--checkpoint-path"
14:"{{.checkpointOutputPrefix}}"
15:"--prev-checkpoint"
16:"{{.prevCheckpointPrefix}}"
17:"--resolver"
18:"flytekit.core.python_auto_container.default_task_resolver"
19:"--"
20:"task-module"
21:""
22:"task-name"
23:"my_task"
]
Expected behavior
The module path is detected and included in the generated command args.
Additional context to reproduce
To reproduce, run the following python script
import flytekit
import flytekit.remote
import flytekit.configuration as flyte_config
from flytekit.tools import repo
import uuid
@flytekit.task
def my_task():
print('my task')
def main(remote_endpoint='dns:///localhost:8089'):
remote = flytekit.remote.FlyteRemote(
config=flyte_config.Config.for_endpoint(endpoint=remote_endpoint, insecure=True),
default_project="flytesnacks",
default_domain="development",
)
detected_root = repo.find_common_root(["."])
print(f"detected root: {detected_root}")
_, native_url = remote.fast_package(detected_root)
fast_serialization_settings = flyte_config.FastSerializationSettings(
enabled=True,
destination_dir=".",
distribution_location=native_url,
)
task = remote.register_task(
my_task,
version=str(uuid.uuid4()).lower().replace("-", ""),
serialization_settings=flyte_config.SerializationSettings(
image_config=flyte_config.ImageConfig.auto_default_image(),
fast_serialization_settings=fast_serialization_settings,
),
)
print(task)
if __name__ == "__main__":
main()
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/14/2023, 4:50 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flytekit/commit/57bf317594bbc84f121beac1571674d8b3500bac|57bf3175>
- Add classifier for Python 3.11 (#1589)
flyteorg/flytekitGitHub
04/14/2023, 5:01 PMflytectl delete execution
seems to be only for terminating executions, I haven't found any other delete option in the UI / APIs.
If the sensitive information is in the raw data storage, I can remove the storage objects directly (but this is cumbersome and prevents you from using primitives).
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/14/2023, 5:52 PM6~8
seconds to 0.25~0.30
seconds.
It can improve performance up to 30x when running a simple workflow.
@task
def demo_task():
print("foobar")
@workflow
def demo_workflow():
demo_task()
Before:
image▾
image▾
lazy_import
in the flytekit to load module lazily.
Tracking Issue
https://flyte-org.slack.com/archives/CP2HDHKE1/p1681239386271109?thread_ts=1681235972.845179&cid=CP2HDHKE1
Follow-up issue
NA
flyteorg/flytekit
GitHub Actions: build (windows-latest, 3.11)
GitHub Actions: build (windows-latest, 3.9)
GitHub Actions: build (windows-latest, 3.8)
GitHub Actions: build (ubuntu-latest, 3.11)
GitHub Actions: build (ubuntu-latest, 3.10)
GitHub Actions: build (ubuntu-latest, 3.9)
GitHub Actions: build (ubuntu-latest, 3.8)
GitHub Actions: Docs Warnings
✅ 2 other checks have passed
2/10 successful checksGitHub
04/14/2023, 6:34 PMGitHub
04/14/2023, 6:58 PMFlyteFile
and am using it within a dataclass to run map tasks. What I've found, is that when converting the type, rather than use my classes to_literal
method, it is converting the custom type to a struct. This makes it act like the normal FlyteFile
rather than my custom type.
Expected behavior
The custom type's to_literal method should be used.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/14/2023, 9:42 PMGitHub
04/14/2023, 9:56 PM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by jsonporter
<https://github.com/flyteorg/flyteconsole/commit/5aecd5ca6a09684b8d41eaa3cefdeb1d54d191b6|5aecd5ca>
- chore: add fallback to task execution link (#743)
flyteorg/flyteconsoleGitHub
04/14/2023, 9:57 PMcache_regions=True
makes it so that aiobotocore
does not hit the access denied bug.
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
flyteorg/flytekit
DCO: DCO
✅ 1 other check has passed
1/2 successful checksGitHub
04/15/2023, 12:02 AM<https://github.com/flyteorg/flyteconsole/tree/master|master>
by ursucarina
<https://github.com/flyteorg/flyteconsole/commit/33ee52a6d3c915535a04800175a5e80b04d3f5e9|33ee52a6>
- chore: allow custom subnav (#734)
flyteorg/flyteconsoleGitHub
04/15/2023, 3:53 AM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/6e4d34c6f1f2b032023befcdc97d46a04ccdc72b|6e4d34c6>
- Pyflyte build imageSpec (#1555)
flyteorg/flytekitGitHub
04/15/2023, 12:10 PMexample_container.py
import logging
import os
from flytekit import ContainerTask, kwtypes, task, workflow
logger = logging.getLogger(__file__)
calculate_ellipse_area_python = ContainerTask(
name="ellipse-area-metadata-python",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(a=float, b=float),
outputs=kwtypes(area=float, metadata=str),
image="<http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2>",
command=[
"python",
"calculate-ellipse-area.py",
"{{.inputs.a}}",
"{{.inputs.b}}",
"/var/outputs"
],
)
@task
def report_all_calculated_areas(
area_python: float,
metadata_python: str
):
<http://logger.info|logger.info>(f"python: area={area_python}, metadata={metadata_python}")
@workflow
def wf(a: float, b: float):
area_python, metadata_python = calculate_ellipse_area_python(a=a, b=b)
# Report on all results in a single task to simplify comparison
report_all_calculated_areas(
area_python=area_python,
metadata_python=metadata_python
)
if __name__ == "__main__":
wf(a=3.0, b = 4.0)
and
calculate-ellipse-area.py
import math
import sys
import os
def write_output(output_dir, output_file, v):
if os.path.isdir(f"{output_dir}") == False:
os.makedirs(f"{output_dir}")
with open(f"{output_dir}/{output_file}", "w") as f:
f.write(str(v))
def calculate_area(a, b):
return math.pi * a * b
def main(a, b, output_dir):
if os.path.exists('var/inputs') == False:
os.makedirs('var/inputs')
a = float(a)
b = float(b)
area = calculate_area(a, b)
write_output(output_dir, "area", area)
write_output(output_dir, "metadata", "[from python rawcontainer]")
if __name__ == "__main__":
a = sys.argv[1]
b = sys.argv[2]
output_dir = sys.argv[3]
main(a, b, output_dir)
Then I tried to run pyflyte
but got the following error:
$ pyflyte run example_container.py wf --a 2.0 --b 3.0
{"asctime": "2023-04-15 12:54:36,959", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
{'a': 2.0, 'b': 3.0}
docker run --rm -v /tmp/inputs:/var/inputs -v /tmp/outputs:/var/outputs <http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2> ['python', 'calculate-ellipse-area.py', '{{.inputs.a}}', '{{.inputs.b}}', '/var/outputs'] None
Failed with Unknown Exception <class 'TypeError'> Reason: 'NoneType' object is not iterable
'NoneType' object is not iterable
Expected behavior
Not to error
Additional context to reproduce
I also tried running example_container.py
directly with Python, and got:
$ python example_container.py
{"asctime": "2023-04-15 12:54:48,793", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
{'a': 3.0, 'b': 4.0}
docker run --rm -v /tmp/inputs:/var/inputs -v /tmp/outputs:/var/outputs <http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2> ['python', 'calculate-ellipse-area.py', '{{.inputs.a}}', '{{.inputs.b}}', '/var/outputs'] None
Traceback (most recent call last):
File "/path/to/flyte/flyte-projects/example/example_container.py", line 44, in <module>
wf(a=3.0, b = 4.0)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 263, in __call__
return flyte_entity_call_handler(self, *args, **input_kwargs)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/promise.py", line 1087, in flyte_entity_call_handler
result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 282, in local_execute
function_outputs = self.execute(**kwargs)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 722, in execute
return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
File "/path/to/env/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
return wrapped(*args, **kwargs)
File "/path/to/flyte/flyte-projects/example/example_container.py", line 35, in wf
area_python, metadata_python = calculate_ellipse_area_python(a=a, b=b)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 299, in __call__
return flyte_entity_call_handler(self, *args, **kwargs) # type: ignore
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/promise.py", line 1079, in flyte_entity_call_handler
return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 280, in local_execute
outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 346, in sandbox_execute
return self.dispatch_execute(ctx, input_literal_map)
File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 556, in dispatch_execute
expected_output_names[i]: native_outputs[i] for i, _ in enumerate(native_outputs)
TypeError: 'NoneType' object is not iterable
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/16/2023, 6:29 AMfrom datetime import datetime
from flytekit import task, workflow, LaunchPlan, CronSchedule
@task
def tk(k: datetime) -> str:
return str(k)
@workflow
def example_workflow(kickoff_time: datetime) -> str:
return tk(k=kickoff_time)
example_launch_plan = LaunchPlan.get_or_create(
name="example_schedule",
workflow=example_workflow,
schedule=CronSchedule(
schedule="*/5 * * * *", # Following schedule runs every min
kickoff_time_input_arg="kickoff_time", # macro magic
),
)
OR
from datetime import datetime
from flytekit import task, workflow, LaunchPlan, CronSchedule
@task
def tk2() -> str:
return "Hello"
@workflow
def example_workflow2() -> str:
return tk2()
example_launch_plan = LaunchPlan.get_or_create(
name="example_schedule2",
workflow=example_workflow2,
schedule=CronSchedule(
schedule="*/5 * * * *", # Following schedule runs every min
),
)
Where the kickoff time arguments are either present or absent.
Expected behavior
It should work in either case
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/16/2023, 6:30 AMGitHub
04/16/2023, 4:48 PM<https://github.com/flyteorg/flyteadmin/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flyteadmin/commit/1026231b8512dc38859816a401413dbd3152f9d1|1026231b>
- Enrich TerminateExecution error to tell propeller the execution already terminated (#551)
flyteorg/flyteadminGitHub
04/16/2023, 5:06 PM<https://github.com/flyteorg/flytepropeller/tree/master|master>
by EngHabu
<https://github.com/flyteorg/flytepropeller/commit/6cb7322ef2ded78ef19200729ab02748c7fdebc1|6cb7322e>
- Check for TerminateExecution error and eat Precondition status (#553)
flyteorg/flytepropellerGitHub
04/16/2023, 5:31 PMGitHub
04/16/2023, 5:43 PMGitHub
04/17/2023, 8:40 AMannotations
configuration to the Vault Secret Manager to provide better flexibility when integrating with Vault.
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☑︎ Smoke tested
☑︎ Unit tests added
☑︎ Code documentation added
☑︎ Any pending items have an associated Issue
Complete description
When the Vault secret manager is enabled, flyte-pod-webhook is responsible for adding vault-specific annotation to the task pod that requests secret variables. These annotations are used to configure the Vault Agent behavior when retrieving and populating the secret from Vault. As of now, the annotation provided by flyte-pod-webhook
is limited to the one implemented in vault_secret_manager.go
.
This PR allows Flyte administrator to add additional or override default annotations providing better flexibility to customize Vault agent behavior. For example, the following configuration will set the Vault agent authentication type to use GCP and to immediately exit if the authentication failed.
vaultSecretManager:
kvVersion: "2"
role: flyte
annotations:
<http://vault.hashicorp.com/auth-config-type|vault.hashicorp.com/auth-config-type> : "gce"
<http://vault.hashicorp.com/auth-type|vault.hashicorp.com/auth-type> : "gcp"
<http://vault.hashicorp.com/agent-auto-auth-exit-on-err|vault.hashicorp.com/agent-auto-auth-exit-on-err> : "true"
Tracking Issue
NA
Follow-up issue
NA
flyteorg/flytepropeller
GitHub Actions: Build & Push Flytepropeller Image
GitHub Actions: Goreleaser
GitHub Actions: Bump Version
✅ 11 other checks have passed
11/14 successful checksGitHub
04/17/2023, 12:20 PMGitHub
04/17/2023, 2:21 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by cosmicBboy
<https://github.com/flyteorg/flytekit/commit/e865db57d3bfbb7fb997417b052a05bc871cb0ed|e865db57>
- Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582)
flyteorg/flytekitGitHub
04/17/2023, 3:36 PM<https://github.com/flyteorg/flyteplugins/tree/master|master>
by hamersaw
<https://github.com/flyteorg/flyteplugins/commit/f5f41825eb28c804a850caad9ba3357a3e1d41d7|f5f41825>
- Override primary container name instead of flyte generated name (#340)
flyteorg/flytepluginsGitHub
04/17/2023, 3:37 PMGitHub
04/17/2023, 3:39 PMpartial
for tasks, which makes writing map tasks a lot more ergonomic. This section of the docs needs to be updated to use the new partial syntax
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
04/17/2023, 4:00 PMGitHub
04/17/2023, 4:00 PM