1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/435436bd112bf7869c83bd5e951ab2d9cf5166ca|435436bd>

- Set PrimaryContainerKey annotation by default (#337) flyteorg/flyteplugins

Release - v1.0.49 New release published by flyte-bot Changelog • 435436b Set PrimaryContainerKey annotation by default (#337) flyteorg/flyteplugins

#555 Setting primaryContainerName by default on Pod plugin Pull request opened by hamersaw TL;DR Pulls the changes from this PR into flytepropeller to reduce Flyte overhead on k8s Pod executions. Basically, we rarely need to wait for the entire`Pod` to complete, rather once the primary container (or all containers) complete we can prematurely mark the task as successful. This mitigates time for the

Pod

operator to detect container completion and mark the

Pod

as completed. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Goreleaser GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

Release - v1.1.81 New release published by flyte-bot Changelog • 0c982ea Remove BarrierTick (#545) flyteorg/flytepropeller

#3595 [BUG] Issue created by Nan2018 Describe the bug When registering flyte entities using the remote API in the same python module, the module path is missing from the generated command args (see below line 21). And executing the entity leads to

ValueError: Empty module name

args:[24 items
0:"pyflyte-fast-execute"
1:"--additional-distribution"
2:"s3://...."
3:"--dest-dir"
4:"{{ .dest_dir }}"
5:"--"
6:"pyflyte-execute"
7:"--inputs"
8:"{{.input}}"
9:"--output-prefix"
10:"{{.outputPrefix}}"
11:"--raw-output-data-prefix"
12:"{{.rawOutputDataPrefix}}"
13:"--checkpoint-path"
14:"{{.checkpointOutputPrefix}}"
15:"--prev-checkpoint"
16:"{{.prevCheckpointPrefix}}"
17:"--resolver"
18:"flytekit.core.python_auto_container.default_task_resolver"
19:"--"
20:"task-module"
21:""
22:"task-name"
23:"my_task"
]

Expected behavior The module path is detected and included in the generated command args. Additional context to reproduce To reproduce, run the following python script

import flytekit
import flytekit.remote
import flytekit.configuration as flyte_config
from flytekit.tools import repo
import uuid

@flytekit.task
def my_task():
    print('my task')


def main(remote_endpoint='dns:///localhost:8089'):
    remote = flytekit.remote.FlyteRemote(
        config=flyte_config.Config.for_endpoint(endpoint=remote_endpoint, insecure=True),
        default_project="flytesnacks",
        default_domain="development",
    )
    detected_root = repo.find_common_root(["."])
    print(f"detected root: {detected_root}")
    _, native_url = remote.fast_package(detected_root)
    fast_serialization_settings = flyte_config.FastSerializationSettings(
        enabled=True,
        destination_dir=".",
        distribution_location=native_url,
    )
    task = remote.register_task(
        my_task,
        version=str(uuid.uuid4()).lower().replace("-", ""),
        serialization_settings=flyte_config.SerializationSettings(
            image_config=flyte_config.ImageConfig.auto_default_image(),
            fast_serialization_settings=fast_serialization_settings,
        ),
    )
    print(task)


if __name__ == "__main__":
    main()

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytekit/commit/57bf317594bbc84f121beac1571674d8b3500bac|57bf3175>

- Add classifier for Python 3.11 (#1589) flyteorg/flytekit

#3596 [Core feature] Deleting executions Issue created by tomasz-sodzawiczny Motivation: Why do you think this is important? When we process customer data in Flyte we need to have the ability to completely remove the related artifacts from the system - either directly after processing, or later at request of the customer. It seems execution artifact deletion came up before: • a use-case was mentioned slack was removing an execution with accidentally leaked sensitive information in the parameters (thread). • #2832 mentions TTL for compliance reasons Goal: What should the final outcome look like, ideally? I would love to be able to: • delete executions one by one • ideally delete groups of executions by something like label or name prefix (e.g. to delete all executions related to a specific customer). Describe alternatives you've considered

flytectl delete execution

seems to be only for terminating executions, I haven't found any other delete option in the UI / APIs. If the sensitive information is in the raw data storage, I can remove the storage objects directly (but this is cumbersome and prevents you from using primitives). Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1590 Lazy load modules Pull request opened by pingsutw TL;DR The loading time can be reduced from

6~8

seconds to

0.25~0.30

seconds. It can improve performance up to 30x when running a simple workflow.

@task
def demo_task():
    print("foobar")


@workflow
def demo_workflow():
    demo_task()

Before: After: Type ☑︎ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description use

lazy_import

in the flytekit to load module lazily. Tracking Issue https://flyte-org.slack.com/archives/CP2HDHKE1/p1681239386271109?thread_ts=1681235972.845179&cid=CP2HDHKE1 Follow-up issue NA flyteorg/flytekit GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: Docs Warnings ✅ 2 other checks have passed 2/10 successful checks

#1591 Add debugpy auto start with `FLYTE_INTERNAL_START_DEBUGPY` Pull request opened by ggydush TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit GitHub Actions: Docs Warnings GitHub Actions: docs DCO: DCO ✅ 1 other check has passed 1/4 successful checks

#3597 [BUG] Flyte DataclassTransformer does not properly use children of FlyteFile Issue created by ggydush Describe the bug I have a custom type that inherits from

FlyteFile

and am using it within a dataclass to run map tasks. What I've found, is that when converting the type, rather than use my classes

to_literal

method, it is converting the custom type to a struct. This makes it act like the normal

FlyteFile

rather than my custom type. Expected behavior The custom type's to_literal method should be used. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#743 chore: add fallback to task execution link Pull request opened by ursucarina TL;DR ContextPanel Allow external hooks to have a fallback header Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Added a fallback option flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release GitHub Actions: lint_project GitHub Actions: unit_tests_with_coverage ✅ 4 other checks have passed 4/9 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/5aecd5ca6a09684b8d41eaa3cefdeb1d54d191b6|5aecd5ca>

- chore: add fallback to task execution link (#743) flyteorg/flyteconsole

#1592 Set cache_regions=True in the case of s3fs Pull request opened by eapolinario TL;DR Set cache_regions to circumvent an aiobotocore bug Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description According to fsspec/s3fs#701 (comment), setting

cache_regions=True

makes it so that

aiobotocore

does not hit the access denied bug. Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit DCO: DCO ✅ 1 other check has passed 1/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by ursucarina

<https://github.com/flyteorg/flyteconsole/commit/33ee52a6d3c915535a04800175a5e80b04d3f5e9|33ee52a6>

- chore: allow custom subnav (#734) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/6e4d34c6f1f2b032023befcdc97d46a04ccdc72b|6e4d34c6>

- Pyflyte build imageSpec (#1555) flyteorg/flytekit

#3598 [BUG] TypeError: 'NoneType' object is not iterable when running a documentation example Issue created by raisadz Describe the bug Hi, I just tried to reproduce an example from the documentation (https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html). I have the following files:

example_container.py

import logging
import os

from flytekit import ContainerTask, kwtypes, task, workflow

logger = logging.getLogger(__file__)


calculate_ellipse_area_python = ContainerTask(
    name="ellipse-area-metadata-python",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(a=float, b=float),
    outputs=kwtypes(area=float, metadata=str),
    image="<http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2>",
    command=[
        "python",
        "calculate-ellipse-area.py",
        "{{.inputs.a}}",
        "{{.inputs.b}}",
        "/var/outputs"
    ],
)

@task
def report_all_calculated_areas(
    area_python: float,
    metadata_python: str
):
    <http://logger.info|logger.info>(f"python: area={area_python}, metadata={metadata_python}")


@workflow
def wf(a: float, b: float):
    area_python, metadata_python = calculate_ellipse_area_python(a=a, b=b)

    # Report on all results in a single task to simplify comparison
    report_all_calculated_areas(
        area_python=area_python,
        metadata_python=metadata_python
    )

if __name__ == "__main__":
    wf(a=3.0, b = 4.0)

and

calculate-ellipse-area.py

import math
import sys
import os


def write_output(output_dir, output_file, v):
    if os.path.isdir(f"{output_dir}") == False:
        os.makedirs(f"{output_dir}")
    with open(f"{output_dir}/{output_file}", "w") as f:
        f.write(str(v))


def calculate_area(a, b):
    return math.pi * a * b


def main(a, b, output_dir):
    if os.path.exists('var/inputs') == False:
        os.makedirs('var/inputs')
    a = float(a)
    b = float(b)

    area = calculate_area(a, b)

    write_output(output_dir, "area", area)
    write_output(output_dir, "metadata", "[from python rawcontainer]")


if __name__ == "__main__":
    a = sys.argv[1]
    b = sys.argv[2]
    output_dir = sys.argv[3]

    main(a, b, output_dir)

Then I tried to run

pyflyte

but got the following error:

$ pyflyte run example_container.py wf --a 2.0 --b 3.0
{"asctime": "2023-04-15 12:54:36,959", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
{'a': 2.0, 'b': 3.0}

docker run --rm -v /tmp/inputs:/var/inputs -v /tmp/outputs:/var/outputs <http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2> ['python', 'calculate-ellipse-area.py', '{{.inputs.a}}', '{{.inputs.b}}', '/var/outputs'] None
Failed with Unknown Exception <class 'TypeError'> Reason: 'NoneType' object is not iterable
'NoneType' object is not iterable

Expected behavior Not to error Additional context to reproduce I also tried running

example_container.py

directly with Python, and got:

$ python example_container.py
{"asctime": "2023-04-15 12:54:48,793", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
{'a': 3.0, 'b': 4.0}

docker run --rm -v /tmp/inputs:/var/inputs -v /tmp/outputs:/var/outputs <http://ghcr.io/flyteorg/rawcontainers-python:v2|ghcr.io/flyteorg/rawcontainers-python:v2> ['python', 'calculate-ellipse-area.py', '{{.inputs.a}}', '{{.inputs.b}}', '/var/outputs'] None
Traceback (most recent call last):
  File "/path/to/flyte/flyte-projects/example/example_container.py", line 44, in <module>
    wf(a=3.0, b = 4.0)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 263, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/promise.py", line 1087, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 282, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/workflow.py", line 722, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "/path/to/flyte/flyte-projects/example/example_container.py", line 35, in wf
    area_python, metadata_python = calculate_ellipse_area_python(a=a, b=b)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 299, in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/promise.py", line 1079, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 280, in local_execute
    outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 346, in sandbox_execute
    return self.dispatch_execute(ctx, input_literal_map)
  File "/path/to/env/lib/python3.10/site-packages/flytekit/core/base_task.py", line 556, in dispatch_execute
    expected_output_names[i]: native_outputs[i] for i, _ in enumerate(native_outputs)
TypeError: 'NoneType' object is not iterable

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3599 [BUG] [flytekit-bug] `pyflyte backfill` does not work with custom kickoff time arguments Issue created by kumare3 Describe the bug For an example like this

from datetime import datetime
from flytekit import task, workflow, LaunchPlan, CronSchedule

@task
def tk(k: datetime) -> str:
  return str(k)


@workflow
def example_workflow(kickoff_time: datetime) -> str:
	return tk(k=kickoff_time)


example_launch_plan = LaunchPlan.get_or_create(
    name="example_schedule",
    workflow=example_workflow,
    schedule=CronSchedule(
        schedule="*/5 * * * *",  # Following schedule runs every min
        kickoff_time_input_arg="kickoff_time", # macro magic
    ),
)

OR

from datetime import datetime
from flytekit import task, workflow, LaunchPlan, CronSchedule

@task
def tk2() -> str:
  return "Hello"


@workflow
def example_workflow2() -> str:
	return tk2()


example_launch_plan = LaunchPlan.get_or_create(
    name="example_schedule2",
    workflow=example_workflow2,
    schedule=CronSchedule(
        schedule="*/5 * * * *",  # Following schedule runs every min
    ),
)

Where the kickoff time arguments are either present or absent. Expected behavior It should work in either case Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#1593 Backfill fix Pull request opened by kumare3 TL;DR • Backfill was using incorrect arguments • backfill should use the argument that user provides or none at all Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Tracking Issue flyteorg/flyte#3599 flyteorg/flytekit GitHub Actions: build-plugins (3.8, flytekit-onnx-pytorch) GitHub Actions: build-plugins (3.8, flytekit-modin) GitHub Actions: build-plugins (3.8, flytekit-mlflow) GitHub Actions: build-plugins (3.8, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.8, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.8, flytekit-kf-mpi) GitHub Actions: build-plugins (3.8, flytekit-k8s-pod) GitHub Actions: build-plugins (3.8, flytekit-hive) GitHub Actions: build-plugins (3.8, flytekit-greatexpectations) GitHub Actions: build-plugins (3.8, flytekit-envd) GitHub Actions: build-plugins (3.8, flytekit-duckdb) GitHub Actions: build-plugins (3.8, flytekit-dolt) GitHub Actions: build-plugins (3.8, flytekit-deck-standard) GitHub Actions: build-plugins (3.8, flytekit-dbt) GitHub Actions: build-plugins (3.8, flytekit-data-fsspec) GitHub Actions: build-plugins (3.8, flytekit-dask) GitHub Actions: build-plugins (3.8, flytekit-bigquery) GitHub Actions: build-plugins (3.8, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.8, flytekit-aws-batch) GitHub Actions: build-plugins (3.8, flytekit-aws-athena) GitHub Actions: build (windows-latest, 3.11) GitHub Actions: build (windows-latest, 3.9) GitHub Actions: build (windows-latest, 3.8) GitHub Actions: build (ubuntu-latest, 3.11) GitHub Actions: build (ubuntu-latest, 3.10) GitHub Actions: build (ubuntu-latest, 3.9) GitHub Actions: build (ubuntu-latest, 3.8) GitHub Actions: docs GitHub Actions: lint GitHub Actions: Docs Warnings ✅ No checks have passed 0/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flyteadmin/commit/1026231b8512dc38859816a401413dbd3152f9d1|1026231b>

- Enrich TerminateExecution error to tell propeller the execution already terminated (#551) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by EngHabu

<https://github.com/flyteorg/flytepropeller/commit/6cb7322ef2ded78ef19200729ab02748c7fdebc1|6cb7322e>

- Check for TerminateExecution error and eat Precondition status (#553) flyteorg/flytepropeller

Release - v1.1.90 New release published by flyte-bot Changelog • 1026231 Enrich TerminateExecution error to tell propeller the execution already terminated (#551) flyteorg/flyteadmin

Release - v1.1.82 New release published by flyte-bot Changelog • 6cb7322 Check for TerminateExecution error and eat Precondition status (#553) flyteorg/flytepropeller

#556 Implement ability to specify additional/override annotations when using Vault Secret Manager Pull request opened by pradithya TL;DR Add

annotations

configuration to the Vault Secret Manager to provide better flexibility when integrating with Vault. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☑︎ Any pending items have an associated Issue Complete description When the Vault secret manager is enabled, flyte-pod-webhook is responsible for adding vault-specific annotation to the task pod that requests secret variables. These annotations are used to configure the Vault Agent behavior when retrieving and populating the secret from Vault. As of now, the annotation provided by

flyte-pod-webhook

is limited to the one implemented in

vault_secret_manager.go

. This PR allows Flyte administrator to add additional or override default annotations providing better flexibility to customize Vault agent behavior. For example, the following configuration will set the Vault agent authentication type to use GCP and to immediately exit if the authentication failed.

vaultSecretManager:
        kvVersion: "2"
        role: flyte
        annotations:
          <http://vault.hashicorp.com/auth-config-type|vault.hashicorp.com/auth-config-type> : "gce"
          <http://vault.hashicorp.com/auth-type|vault.hashicorp.com/auth-type> : "gcp"
          <http://vault.hashicorp.com/agent-auto-auth-exit-on-err|vault.hashicorp.com/agent-auto-auth-exit-on-err> : "true"

Tracking Issue NA Follow-up issue NA flyteorg/flytepropeller GitHub Actions: Build & Push Flytepropeller Image GitHub Actions: Goreleaser GitHub Actions: Bump Version ✅ 11 other checks have passed 11/14 successful checks

#3601 auto-update contributors Pull request opened by flyte-bot Automated changes by create-pull-request GitHub action flyteorg/flyte ✅ All checks have passed 6/6 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytekit/commit/e865db57d3bfbb7fb997417b052a05bc871cb0ed|e865db57>

- Feature/3506: Improve Type Conversion errors, use rich to prettify error messages (#1582) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flyteplugins/commit/f5f41825eb28c804a850caad9ba3357a3e1d41d7|f5f41825>

- Override primary container name instead of flyte generated name (#340) flyteorg/flyteplugins

Release - v1.0.50 New release published by flyte-bot Changelog • f5f4182 Override primary container name instead of flyte generated name (#340) flyteorg/flyteplugins

#3603 [Docs] Update map task docs for how to handle default args with partial Issue created by cosmicBboy Description flytekit 1.5.0 added support for using

partial

for tasks, which makes writing map tasks a lot more ergonomic. This section of the docs needs to be updated to use the new partial syntax Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Pending review on flyteorg/flytekit-java - Manage reminder [#6] Update .env.local in README (kanterov) a year stale · 3 years old · Waiting on _ @Ketan (kumare3)_

Pending review on flyteorg/flyteplugins - Manage reminder [#330] External Plugin Service (grpc) (pingsutw) 3 days stale · a month old · Waiting on _ hamersaw_