Release - v1.1.104 New release published by flyte-bot Changelog • 0cfb0a0 Added IsFailurePermanent flag on DynamicTaskStatus (#567) flyteorg/flytepropeller

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/0194eb6dd6438410f64fee8520e0cf12a45c2fbc|0194eb6d>

- cleanup docker build, debug flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/b23ef278d74d5438cf168d14fde4705e39798dfd|b23ef278>

- debugging flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/ea0d7b04f8c125110fee1a8572c376911e1515b1|ea0d7b04>

- debugging flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/ccfab5b3da86323f07a643ab576d0ad3ed37e3ea|ccfab5b3>

- debugging flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by cosmicBboy

<https://github.com/flyteorg/flytekit/commit/ca1608e5dcfb4d84dd9b4fc9c206ad6e4f4e825c|ca1608e5>

- use getattr to access task_def.disable_deck in entrypoint (#1724) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/f52526d47aa3175c021da84cbf08a97fc7da7dd8|f52526d4>

- Add a couple tests (#1722) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flyte/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyte/commit/b46fbd28dfa4b71f0131063a757560c81b30460b|b46fbd28>

- Fix single binary config (#3808) flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/fdd7a6a643575ce52e66bd5eecd8f727364e5d21|fdd7a6a6>

- end to end tests pass flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flytectl/tree/master|master>

by katrogan

<https://github.com/flyteorg/flytectl/commit/47d6c6255e44b1b618c07575b2cbe78f3787aab3|47d6c625>

- Fix the error message for the update cmd (#411) flyteorg/flytectl

Release - v0.6.41 New release published by flyte-bot Changelog • 47d6c62 Fix the error message for the update cmd (#411) flyteorg/flytectl

1 new commit pushed to

<https://github.com/flyteorg/homebrew-tap/tree/main|main>

by flyte-bot

<https://github.com/flyteorg/homebrew-tap/commit/5011a7e3d356977e95b6efefd2071d925c66b64b|5011a7e3>

- Brew formula update for flytectl version v0.6.41 flyteorg/homebrew-tap

#3843 [Core feature] Add grace period before declaring ImagePullBackoff a failure Issue created by hamersaw Motivation: Why do you think this is important? Currently when FlytePropeller detects

ImagePullBackoff

on a

Pod

it immediately declares the task a failure. In most scenarios this may not be the most efficient behavior because FlytePropeller will then retry with a new

Pod

(given retries are enabled) which will just attempt to pull the same image(s). By introducing a configurable grace period before FlytePropeller declares a task failure we allow k8s to continue attempting to pull the image. A grace period mechanism has precedent in other issues as well. Goal: What should the final outcome look like, ideally? FlytePropeller should wait a configurable period before declaring an

ImagePullBackoff

a failure rather than immediately on first detection. Describe alternatives you've considered Leaving this as is causes unnecessary failures / retries. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/26291f192b6760e23d2e6aff418a14dd8c1880a2|26291f19>

- Csvtransform (#1671) flyteorg/flytekit

#1671 Csvtransform Pull request opened by ChungYujoyce Data Type transformation (Pandas <-> CSV) and its unit tests Type ☐ Bug Fix • [*] Feature ☐ Plugin Are all requirements met? • [*] Code completed • [*] Smoke tested • [*] Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description add an encoder and decoder for pandas <-> CSV transformation Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/30060324122760a57345722cea8adcb154875f49|30060324>

- add private registry support (#1705) flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/e3383b3852c4933a1b3fb6a2cc2fbc017e0bc11f|e3383b38>

- updates flyteorg/flyte-conference-talks

1 new commit pushed to

<https://github.com/flyteorg/flyte-conference-talks/tree/main|main>

by cosmicBboy

<https://github.com/flyteorg/flyte-conference-talks/commit/d6714b376e2c56083a1eb1d9fbbe4cd15f24ac43|d6714b37>

- fix notebook flyteorg/flyte-conference-talks

#3845 [BUG] Duplicate Database Creation Issue created by Yicheng-Lu-llll Describe the bug When first running:

flytectl demo start --dev
# Under flyte repo
make compile
POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml

The following errors may occurs.

{"json":{"src":"start.go:63"},"level":"info","msg":"Running Database Migrations...","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"initialize.go:41"},"level":"warning","msg":"Database [flyte] does not exist, trying to create it now","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"database.go:123"},"level":"warning","msg":"Database [flyte] does not exist","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"database.go:136"},"level":"info","msg":"Creating database flyte","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"handle.go:59"},"level":"info","msg":"Creating Database flyte since it does not exist","ts":"2023-07-07T05:13:37Z"}
{"json":{"src":"initialize.go:54"},"level":"error","msg":"Failed to create DB flyte err ERROR: duplicate key value violates unique constraint \"pg_database_datname_index\" (SQLSTATE 23505)","ts":"2023-07-07T05:13:37Z"}
panic: ERROR: duplicate key value violates unique constraint "pg_database_datname_index" (SQLSTATE 23505)

It seems that during initialization, the process attempts to create the flyte database twice. this isn't a big issue as it can be resolved by simply rerunning the command

POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml

. It will not encounter the same issue on subsequent runs as the flyte database will already exist. Expected behavior No errors when first running

POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml

. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#370 Add ImagePullBackoffGracePeriod configuration Pull request opened by hamersaw TL;DR This PR adds a

ImagePullBackoffGracePeriod

configuration option so that Flyte does not automatically declare

Pods

with

ImagePullBackoff

as failures, but rather waits a configurable duration for k8s to recover. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue fixes flyteorg/flyte#3843 Follow-up issue NA flyteorg/flyteplugins ✅ All checks have passed 2/2 successful checks

#109 Fix archive.replacements in goreleaser Pull request opened by eapolinario TL;DR Similar flyteorg/flytepropeller#586 Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/datacatalog ✅ All checks have passed 2/2 successful checks

#61 Update .goreleaser.yml Pull request opened by eapolinario Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytecopilot GitHub Actions: Goreleaser GitHub Actions: Build & Push Flyte copilot Image GitHub Actions: Bump Version ✅ 9 other checks have passed 9/12 successful checks

#158 Update .goreleaser.yml Pull request opened by eapolinario Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytestdlib ✅ All checks have passed 5/5 successful checks

#1725 Airflow agent Pull request opened by pingsutw TL;DR Airflow agent allows you to seamlessly run Airflow tasks in the Flyte workflow without changing code. All the airflow tasks will be run on an airflow agent (long-running server) instead of launching a new pod to run it, which dramatically reduces overhead. • Compile Airflow tasks to Flyte tasks • Use Airflow sensors/operators in Flyte workflows • Add support running Airflow tasks locally without running a cluster Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description DataprocOperator

from datetime import timedelta

from airflow.utils import trigger_rule

from flytekit import task, workflow, ImageSpec
from airflow.sensors import time_sensor
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator, DataprocDeleteClusterOperator, DataprocSubmitSparkJobOperator

airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@workflow
def wf():
    create_cluster = DataprocCreateClusterOperator(
        task_id="create_dataproc_cluster",
        image_version="2.0.27-debian10",
        storage_bucket="opta-gcp-dogfood-gcp",
        master_machine_type="n1-highmem-32",
        master_disk_size=1024,
        num_workers=2,
        worker_machine_type="n1-highmem-64",
        worker_disk_size=1024,
        region="us-west1",
        cluster_name="flyte-dataproc",
        project_id="dogfood-gcp-dataplane",
    )

    run_spark = DataprocSubmitSparkJobOperator(
        job_name="spark_pi",
        task_id="run_spark",
        dataproc_jars=["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
        main_class="org.apache.spark.examples.JavaWordCount",
        arguments=["<gs://opta-gcp-dogfood-gcp/spark/file.txt>"],
        cluster_name="flyte-dataproc",
        region="us-west1",
        project_id="dogfood-gcp-dataplane",
    )

    delete_cluster = DataprocDeleteClusterOperator(
        task_id="create_dataproc_cluster",
        project_id="dogfood-gcp-dataplane",
        cluster_name="flyte-dataproc",
        region="us-west1",
        retries=3,
        retry_delay=timedelta(minutes=5),
        email_on_failure=True,
        trigger_rule=trigger_rule.TriggerRule.ALL_DONE
    )

    create_cluster >> run_spark >> delete_cluster


if __name__ == '__main__':
    wf()

bashSensor

from airflow.sensors.bash import BashSensor
from flytekit import task, workflow, ImageSpec


airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@task(container_image=image_spec)
def t1():
    print("flyte")


@workflow
def wf():
    sensor = BashSensor(task_id="Sensor_succeeds", bash_command="echo hello")
    sensor >> t1()


if __name__ == '__main__':
    wf()

HiveSensor

from airflow.providers.apache.hive.sensors.hive_partition import HivePartitionSensor

from flytekit import task, workflow, ImageSpec


airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@task(container_image=image_spec)
def t1():
    print("flyte")


@workflow
def wf():
    sensor = HivePartitionSensor(table="flyte", schema="person")
    sensor >> t1()


if __name__ == '__main__':
    wf()

FileSensor

from airflow.sensors.filesystem import FileSensor
from flytekit import task, workflow, ImageSpec


airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@task(container_image=image_spec)
def t1():
    print("flyte")


@workflow
def wf():
    sensor = FileSensor(task_id="id", filepath="/tmp/1234")
    sensor >> t1()


if __name__ == '__main__':
    wf()

TimeSensor

from datetime import datetime, timedelta
from pytz import UTC

from airflow.sensors.time_sensor import TimeSensor
from flytekit import task, workflow, ImageSpec

airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@task(container_image=image_spec)
def t1():
    print("flyte")


@workflow
def wf():
    sensor = TimeSensor(task_id="fire_immediately", target_time=(datetime.now(tz=UTC)+timedelta(seconds=5)).time())
    sensor >> t1()


if __name__ == '__main__':
    wf()

PythonSensor

from airflow.sensors.python import PythonSensor
from flytekit import task, workflow, ImageSpec

airflow_plugin = "git+<https://github.com/flyteorg/flytekit.git@487438ab59147879eded897674593a1eaee1c78b#subdirectory=plugins/flytekit-airflow>"
image_spec = ImageSpec(base_image="pingsutw/flytekit:v1", packages=["apache-airflow", airflow_plugin], apt_packages=["git"], registry="pingsutw")


@task(container_image=image_spec)
def t1():
    print("flyte")


def py_func():
    print("airflow python sensor")
    return True


@workflow
def wf():
    sensor = PythonSensor(task_id="fire_immediately", python_callable=py_func)
    sensor >> t1()


if __name__ == '__main__':
    wf()

Tracking Issue NA Follow-up issue NA flyteorg/flytekit GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-snowflake) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.11, flytekit-kf-mpi) GitHub Actions: build-plugins (3.11, flytekit-k8s-pod) GitHub Actions: build-plugins (3.11, flytekit-huggingface) GitHub Actions: build-plugins (3.11, flytekit-hive) GitHub Actions: build-plugins (3.11, flytekit-envd) GitHub Actions: build-plugins (3.11, flytekit-duckdb) GitHub Actions: build-plugins (3.11, flytekit-dolt) GitHub Actions: build-plugins (3.11, flytekit-deck-standard) GitHub Actions: build-plugins (3.11, flytekit-dbt) GitHub Actions: build-plugins (3.11, flytekit-data-fsspec) GitHub Actions: build-plugins (3.11, flytekit-dask) GitHub Actions: build-plugins (3.11, flytekit-bigquery) GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.11, flytekit-aws-batch) GitHub Actions: build-plugins (3.11, flytekit-aws-athena) GitHub Actions: build-plugins (3.10, flytekit-whylogs) GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.10, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.10, flytekit-spark) GitHub Actions: build-plugins (3.10, flytekit-ray) GitHub Actions: build-plugins (3.10, flytekit-polars) GitHub Actions: build-plugins (3.10, flytekit-papermill) ✅ 1 other check has passed 1/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/05981db6cef7f96122e5728963d5a80457ebc023|05981db6>

- Fix archive.replacements in goreleaser (#586) flyteorg/flyteadmin

#3846 [BUG] webUI doesn't show the reason for a registration failure of dynamic tasks Issue created by alekpikl Describe the bug A dynamic task is registered at runtime. If this fails, the web UI doesn't show it. The screenshot shows what I can see in the WebUI. The task is marked as failed, but in the execution tab, it says succeeded. The error must then be retrieved from the Flyte Propeller logs. Expected behavior 1. The error is shown in the web UI 2. The execution tab also shows as failure. Additional context to reproduce No response Screenshots Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

Release - v1.1.110 New release published by flyte-bot Changelog • 05981db Fix archive.replacements in goreleaser (#586) flyteorg/flyteadmin

1 new commit pushed to

<https://github.com/flyteorg/flytestdlib/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytestdlib/commit/53f62a741e0f697f94129e5541d027920f89f115|53f62a74>

- Update .goreleaser.yml (#158) flyteorg/flytestdlib

1 new commit pushed to

<https://github.com/flyteorg/flytecopilot/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flytecopilot/commit/10f74209f0f0141e4f28dbd40a2d378e77c41c28|10f74209>

- Update .goreleaser.yml (#61) flyteorg/flytecopilot

1 new commit pushed to

<https://github.com/flyteorg/datacatalog/tree/master|master>

by eapolinario

<https://github.com/flyteorg/datacatalog/commit/5e2512ab65baf283fafd3dafcde04bf1a1c5a56c|5e2512ab>

- Fix archive.replacements in goreleaser (#109) flyteorg/datacatalog