Release - v1.5.2 New release published by flyte-bot Changelog • 44b6bc6 add user_identifier (#388) flyteorg/flyteidl

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/35e52ef669dcfd682f53c4b2bdbed628838945e1|35e52ef6>

- External Plugin Service (grpc) (#1524) flyteorg/flytekit

#1524 External Plugin Service (grpc) Pull request opened by pingsutw TL;DR RFC Dockerfile for out of core plugin. Deployment yaml for out of core plugin. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue flyteorg/flyte#3282 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks

#3642 [BUG] Single task executions should not default to a node execution deadline of 24 hours Issue created by jeevb Describe the bug It is not clear if this is a bug, or is intended for a good reason. Currently, single task executions receive a default hard coded timeout of 24 hours: https://github.com/flyteorg/flyteadmin/blob/950ad1522aa8697777b86bb39e5df73eaf64e530/pkg/manager/impl/util/single_task_execution.go#L29 As such, they do not respect the default deadlines specified at the platform level, or the timeout specified at the task level in Flytekit. Expected behavior Single task executions do not timeout by default at 24 hours, and respect the task's timeout or the platform node execution deadline. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3643 [BUG] - pyflyte run with custom image option is not using the specified image Issue created by mitzen Describe the bug i am trying to register my workflow to flyte sandbox environment, using the following command:

pyflyte register workflows --image <http://ghcr.io/flyteorg/flytekit:py3.11-sqlalchemy-1.6.0b0|ghcr.io/flyteorg/flytekit:py3.11-sqlalchemy-1.6.0b0>

This image exist https://github.com/flyteorg/flytekit/pkgs/container/flytekit/86821037?tag=py3.11-sqlalchemy-1.6.0b0 I can see that the workflow is registered. However, when i look at the image, it is defaulted to

<http://cr.flyte.org/flyteorg/flytekit:py3.11-sqlalchemy-1.5.0|cr.flyte.org/flyteorg/flytekit:py3.11-sqlalchemy-1.5.0>

might have been a parsing error here, missing out gr in cr.flyte.org/flyteorg/flytekit:py3.11-sqlalchemy-1.5.0 Please have a look at the screenshot below:- Expected behavior workflow should be using the image given in the cli Additional context to reproduce 1. Use the sample code that was given here https://raw.githubusercontent.com/flyteorg/flytesnacks/master/cookbook/integrations/flytekit_plugins/sql/sql_alchemy.py 2. Then try to register the workflow using "pyflyte register workflows --image ghcr.io/flyteorg/flytekit:py3.11-sqlalchemy-1.6.0b" 3. Go to the web ui and look at the task details, expand the tree, goto -> container -> image name Screenshots Registration was made for image ghcr.io/flyteorg/flytekit:py3.11-sqlalchemy-1.6.0b0 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3644 [BUG] Subworkflow chaining does not work when run locally Issue created by d-tw Describe the bug When the order of two subworkflows is controlled via the chaining operator, the order of execution is not applied when the code is run locally (locally as in "via direct invocation of python", not locally as in "via a local cluster"). When the workflow is published to Flyte, the DAG and subsequent execution are correct. Expected behavior The order of execution of subworkflows should be identical between local runs and remote execution, especially in the case of unit testing. Additional context to reproduce The version of flytekit is 1.5.0. Sample code:

from flytekit import workflow, task
import logging

logging.basicConfig(level="DEBUG")

logger = logging.getLogger("wf")
logger.setLevel("DEBUG")

@task
def task_1():
    <http://logger.info|logger.info>("task_1")

@task
def task_2():
    <http://logger.info|logger.info>("task_2")

@workflow
def sub_workflow_1():
    task_1()

@workflow
def sub_workflow_2():
    task_2()

@workflow
def parent_workflow():
    s1 = sub_workflow_1()
    s2 = sub_workflow_2()

    s2 >> s1

if __name__ == "__main__":
    parent_workflow()

Output is:

INFO:wf:task_1
INFO:wf:task_2

Expected output is:

INFO:wf:task_2
INFO:wf:task_1

Screenshots When registered with a Flyte cluster, the DAG is correct. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3645 [BUG] No kubeconfig for flytectl demo cluster Issue created by tekumara Describe the bug

❯ flytectl demo start 
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
W0507 10:37:51.431316   66838 loader.go:221] Config not found: /home/compute/.flyte/k3s/k3s.yaml
🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
🐋 Going to use Flyte v1.5.0 release with image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bc521dac6f0dc6e1942efa6447611cd3354540c4|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bc521dac6f0dc6e1942efa6447611cd3354540c4> 
🐋 pulling docker image for release <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bc521dac6f0dc6e1942efa6447611cd3354540c4|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-bc521dac6f0dc6e1942efa6447611cd3354540c4>
🧑‍🏭 booting Flyte-sandbox container
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
Waiting for cluster to come up...
W0507 10:38:10.078019   66838 loader.go:221] Config not found: /home/compute/.flyte/k3s/k3s.yaml
W0507 10:38:10.078495   66838 loader.go:221] Config not found: /home/compute/.flyte/k3s/k3s.yaml
Error: open /home/compute/.flyte/k3s/k3s.yaml.lock: permission denied
{"json":{},"level":"error","msg":"open /home/compute/.flyte/k3s/k3s.yaml.lock: permission denied","ts":"2023-05-07T10:38:10Z"}

❯ cat ~/.flyte/k3s/k3s.yaml
cat: /home/compute/.flyte/k3s/k3s.yaml: No such file or directory

❯ ll ~/.flyte/k3s/
total 0
lrwxrwxrwx 1 root root 32 May  7 10:38 k3s.yaml -> /var/lib/flyte/config/kubeconfig

Expected behavior ~/.flyte/k3s/k3s.yaml is created by

flytectl demo start

Additional context to reproduce The host is ubuntu linux

❯ flytectl version
{
  "App": "flytectl",
  "Build": "bd6b856",
  "Version": "0.6.36",
  "BuildTime": "2023-05-07 10:43:00.839982826 +0000 UTC m=+0.029162480"
}%

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#557 Add RateLimit middleware using TokenBucket algorithm Pull request opened by LaPetiteSouris TL;DR Solve flyteorg/flyte#327 Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Main changes • Add a simple rate limiter middleware. • The RateLimiter components rely on an in-memory map storage to store

userID

and

rate

per user ID.

rate

is a native Golang package to track total count of requests. • The in-memory storage is cleaned periodically to reduce memory footprint. • The component, which is considered as part of the security scope, relies on the identity of the user provided by

auth

package. In fact, the best way to uniquely identify requests to rate limit is to track usage per authenticated user. Tradeoff • To keep it simple, an in-memory map storage is used to track rate per user. This becomes inaccurate if multiple instances of

flyteadmin

is deployed. If this is the case and we are serious about rate limit, another improvement is needed, like introduction of Redis for example, which is out of scopes for this PR. Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes flyteorg/flyte#327 Follow-up issue NA flyteorg/flyteadmin ✅ All checks have passed 2/2 successful checks

Pending review on flyteorg/flytekit-java - Manage reminder [#6] Update .env.local in README (kanterov) a year stale · 3 years old · Waiting on _ @Ketan (kumare3)_

Pending review on flyteorg/flytekit-python-template - Manage reminder [#33] Integration testing suite v1 (zeryx) 3 days stale · 4 days old · Waiting on _ eapolinario, @Niels Bantilan_

1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/e28a142e966cf689fd8b51ba9049763626dc2eba|e28a142e>

- Add support fetching description entity (#735) flyteorg/flyteconsole

#735 Add support fetching description entity Pull request opened by pingsutw TL;DR • Fetch / list description entity • Add short description to SearchableTaskNameList • Add short description to SearchableWorkflowNameList • Add long description and source link to EntityDetails Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Tracking Issue NA Follow-up issue NA flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 8 other checks have passed 8/11 successful checks

#3648 [BUG] pyspark pipeline type transformer error Issue created by peridotml Describe the bug The pyspark

PipelineModel

transformer works when running spark locally, but is failing to load inside the k8s spark plugin. This is likely because the transformer downloads the pipeline to the driver and it is NOT available on the workers. Likely we need to save and load directly to the remote path. Expected behavior The type transformer shouldn't fail. Additional context to reproduce

from flytekit import workflow, task, StructuredDataset
from flytekitplugins.spark import Spark
import flytekit
import pandas as pd
from typing import Tuple

from pyspark.ml.feature import StringIndexer
from pyspark.ml.pipeline import Pipeline, PipelineModel


@task(cache=True, cache_version="1.0")
def create_df() -> pd.DataFrame:
    return pd.DataFrame(
        data={
        "id": [1, 2, 3, 4, 5, 6, 7, 8, 9],
        "cat": ["a", "b", "c", "a", "b", "c", "a", "b", "c"],
        "num": [1, 2, 3, 4, 5, 6, 7, 8, 9],
    })


@task(task_config=Spark(
    spark_conf={
            "spark.driver.memory": "4g",
            "spark.executor.memory": "2g",
            "spark.executor.instances": "1",
            "spark.driver.cores": "2",
            "spark.executor.cores": "1",
    }
), cache=True, cache_version="1.0")
def save_pipeline(df: pd.DataFrame) -> Tuple[pd.DataFrame, PipelineModel]:
    spark = flytekit.current_context().spark_session

    spark_df = spark.createDataFrame(df)

    cat_indexer = StringIndexer(inputCol="cat", outputCol="cat_index")
    pipeline = Pipeline(stages=[cat_indexer])

    fitted_pipeline = pipeline.fit(spark_df)
    spark_df_transformed = fitted_pipeline.transform(spark_df)

    return (
        spark_df_transformed.toPandas(),
        fitted_pipeline
    )

@task(task_config=Spark(
    spark_conf={
        "spark.driver.memory": "4g",
        "spark.executor.memory": "2g",
        "spark.executor.instances": "1",
        "spark.driver.cores": "2",
        "spark.executor.cores": "1",
    }
))
def compare_dfs(pipeline: PipelineModel, df: pd.DataFrame, expected: pd.DataFrame):
    spark = flytekit.current_context().spark_session
    spark_df = spark.createDataFrame(df)
    df_transformed = pipeline.transform(spark_df).toPandas()

    dataframes_equal = expected.sort_values(by=["id"]).equals(
        df_transformed.sort_values(by=["id"]).reset_index(drop=True)
    )

    assert dataframes_equal, "Dataframes are not equal"


@workflow
def test_pipeline_transformer():
    df = create_df()
    df_transformed, pipeline = save_pipeline(df=df)
    compare_dfs(pipeline=pipeline, df=df, expected=df_transformed)

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
        native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in literal_map_to_kwargs
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in <dictcomp>
        return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 831, in to_python_value
        return transformer.to_python_value(ctx, lv, expected_python_type)
      File "/opt/venv/lib/python3.9/site-packages/flytekitplugins/spark/pyspark_transformers.py", line 42, in to_python_value
        return PipelineModel.load(local_dir)
      File "/opt/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 353, in load
        return cls.read().load(path)
      File "/opt/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 282, in load
        metadata = DefaultParamsReader.loadMetadata(path, <http://self.sc|self.sc>)
      File "/opt/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 565, in loadMetadata
        metadataStr = sc.textFile(metadataPath, 1).first()
      File "/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1906, in first
        raise ValueError("RDD is empty")

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#753 chore: remove disable_auth from docs Pull request opened by ursucarina TL;DR Removes legacy references to DISABLE_AUTH from readme s Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Removes legacy references to DISABLE_AUTH from readme s Tracking Issue NA flyteorg/flyteconsole Codecov: 60.86% of diff hit (target 66.09%) Codecov: 66.07% (-0.02%) compared to e28a142 GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release ✅ 6 other checks have passed 6/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flytekit/commit/7efde40e1c396ed461ca16958462056f6af09938|7efde40e>

- Include traceback in errors from admin (#1621) flyteorg/flytekit

#1623 Fix PipelineModel transformer issue 3648 Pull request opened by peridotml TL;DR This PR fixes #3648 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/00915ec5bc25b2dbe739767d5e060e988f6402e2|00915ec5>

- Add migration to turn

parent_id

column into

bigint

only if necessary (#554) flyteorg/flyteadmin

#3649 Remove mentions to DISABLE_AUTH Pull request opened by eapolinario Tracking issue Closes #3636 Describe your changes Check all the applicable boxes ☐ I updated the documentation accordingly. ☐ All new and existing tests passed. ☐ All commits are signed-off. Screenshots Note to reviewers flyteorg/flyte ✅ All checks have passed 8/8 successful checks

Release - v1.1.92 New release published by flyte-bot Changelog • 00915ec Add migration to turn

parent_id

column into

bigint

only if necessary (#554) flyteorg/flyteadmin

#3650 Update Flyte components Pull request opened by flyte-bot Updated flyte deployment • Updated GCP Flyte kustomize generated manifest file • Updated EKS Flyte kustomize generated manifest file • Updated Sandbox Flyte kustomize generated manifest file • Updated TEST Flyte kustomize generated manifest file • Updated GCP Flyte helm generated manifest file • Updated EKS Flyte helm generated manifest file • Updated Sandbox Flyte helm generated manifest file • Updated TEST Flyte helm generated manifest file • Auto-generated by [flyte-bot] flyteorg/flyte ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteadmin/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteadmin/commit/507ad4497beefe0e157515617027e5a8a2bd8483|507ad449>

- Added start time for supporting restarts for fixed rate schedules (#476) flyteorg/flyteadmin

#2885 [BUG] [Flytescheduler] Fixed rate scheduler doesn't respect the initial startTime after a restart Issue created by pmahindrakar-oss Describe the bug Fixed rate scheduler allows to schedule flyte executions at defined regular intervals. Rates supported are hour, day and minutes. When scheduler starts and its given a fixed rate schedule then it uses the current time to schedule new executions at an interval provided by fixed rate In case the scheduler restarts , it should use the last known time stamp of the execution if available from the snapshot or use the updatedAt time to start the new fixed rate schedule. Expected behavior Fixed rate scheduler should schedule new executions using the last know executions time or using the updatedAt timestamp of the schedule Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#476 Added start time for supporting restarts for fixed rate schedules Pull request opened by pmahindrakar-oss Signed-off-by: Prafulla Mahindrakar prafulla.mahindrakar@gmail.com TL;DR • Updated the cron library to accept startTime for timedJob • Using the lastExecution time when starting the fixed rate schedules Testing Done: • Registered the fixed rate schedule for 1 min rate • Brought down the scheduler for a few hours • Verified the catch up routine ran and caught up to current time using the last Execution timestamp • Also verified the new routines are also respecting the last execution time Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes flyteorg/flyte#2885 Follow-up issue flyteorg/flyteadmin GitHub Actions: Build & Push Flytescheduler Image GitHub Actions: Goreleaser GitHub Actions: Build & Push Flyteadmin Image GitHub Actions: Bump Version Codecov: 58.39% (-0.25%) compared to 00915ec ✅ 9 other checks have passed 9/14 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/ca4676103d8e5b41e32d507b88d4faaa9d276a6d|ca467610>

- Add support for copying all the files in source root (#1622) flyteorg/flytekit

Release - External Plugin Service New release published by pingsutw #1524 flyteorg/flytekit

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by peridotml

<https://github.com/flyteorg/flytekit/commit/35bb5561537e45a32c6ae328684fe09cea318786|35bb5561>

- fix PipelineModel transformer issue 3648 (#1623) flyteorg/flytekit

#401 Support union and none type in flyteidl Pull request opened by yubofredwang TL;DR Union type is not supported in flyteidl leading to flytectl failures when the task takes Union type as input or output. When users do operations such as:

flytectl get execution

will output:

failed to convert to a known Literal. Input Type [union_type:..] not supported

Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description For creating making literal for union type, the code will find the first one that matches the input value. If the input is 1.0 and union type is Union(Integer, Float), integer will be used since 1.0 will match with integer. For default generation of a union type, I just use the first variants possible. For example, if the union contains Union(String, Integer), then default value generated when creating an execFile will be "" not 0. Tested with flytectl commands, get task and create execution is working fine Tracking Issue fixes flyteorg/flyte#3094 flyteorg/flyteidl ✅ All checks have passed 13/13 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteplugins/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flyteplugins/commit/76a80ec5b7240d16fe5a1a190d51d8b784b88115|76a80ec5>

- Add nil check in databricks plugin (#332) flyteorg/flyteplugins

#332 Add nil check in databricks plugin Pull request opened by pingsutw TL;DR Add nil check in databricks plugin. should fix https://flyte-org.slack.com/archives/CP2HDHKE1/p1678980405562029?thread_ts=1678831311.261609&cid=CP2HDHKE1 Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue ^^^ Follow-up issue NA flyteorg/flyteplugins Codecov: 0.00% of diff hit (target 62.65%) Codecov: 62.64% (-0.02%) compared to 18a594e ✅ 4 other checks have passed 4/6 successful checks

Release - v1.0.57 New release published by flyte-bot Changelog • 76a80ec Add nil check in databricks plugin (#332) flyteorg/flyteplugins