1 new commit pushed to

<https://github.com/flyteorg/flyteconsole/tree/master|master>

by jsonporter

<https://github.com/flyteorg/flyteconsole/commit/8f8d6869e1c2e064b2138d1b0e3f5745feb47849|8f8d6869>

- chore: [tlm] comprehensive node execution query (#749) flyteorg/flyteconsole

1 new commit pushed to

<https://github.com/flyteorg/flytepropeller/tree/master|master>

by hamersaw

<https://github.com/flyteorg/flytepropeller/commit/0025cbea7084089597691d88fb698bd8d7ae45f0|0025cbea>

- Added support for aborting task nodes reported as failures (#541) flyteorg/flytepropeller

Release - v1.1.86 New release published by flyte-bot Changelog • 0025cbe Added support for aborting task nodes reported as failures (#541) flyteorg/flytepropeller

#3629 [UI Feature] Union type is not supported on FlyteConsole Issue created by yubofredwang Motivation: Why do you think this is important? It is a common use case for Flyte users to set Union type for their task input. However, the input value won't show up on the flyte console UI and instead it will show as below Goal: What should the final outcome look like, ideally? The output should show as:

input name: {
   type: float
   value: 0.5
}

Propose: Link/Inline OR Additional context Implement the function here: https://github.com/flyteorg/flyteconsole/blob/8f8d6869e1c2e064b2138d1b0e3f5745feb47849/packages/console/src/components/Literals/helpers.ts#L145 Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3630 [BUG] K8s Spark plugin can't read or write from s3 out of the box Issue created by peridotml Describe the bug A few people (including me) have run into the following error when trying to save a DataFrame from a spark task when using s3 as a file system.

error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o42.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

• https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to. • https://discuss.flyte.org/t/39240/U03CZJFCT7T-you-should-be-able-to-find-pods-in-the-flytesnac Expected behavior We would expect this to work out of the box by providing sensible defaults. I did find a solution. The task works when I include

configure_s3

. Additional context to reproduce

import pandas as pd

from flytekit import task, workflow, StructuredDataset
from flytekitplugins.spark import Spark
import flytekit

def configure_s3(spark):
    hadoop_conf = spark._jsc.hadoopConfiguration()
    hadoop_conf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    hadoop_conf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.WebIdentityTokenCredentialsProvider")


@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:k8s_spark-latest|ghcr.io/flyteorg/flytecookbook:k8s_spark-latest>", task_config=Spark(
        spark_conf={
            "spark.driver.memory": "1000M",
            "spark.executor.memory": "1000M",
            "spark.executor.instances": "3",
            "spark.driver.cores": "1",
            "spark.executor.cores": "1",
        }),
)
def t1() -> StructuredDataset:
    spark = flytekit.current_context().spark_session
    
    # The task fails without these settings.
    # configure_s3(spark)

    df = pd.DataFrame({"a": [1,2,3]})
    df = spark.createDataFrame(df)
    return StructuredDataset(dataframe=df)

@workflow
def wf() -> StructuredDataset:
    sd = t1()
    return sd

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by kumare3

<https://github.com/flyteorg/flytekit/commit/4ad64d3037cc6227019c489b13e29c3aef21bb74|4ad64d30>

- Add is_inside func to ImageSpec (#1601) flyteorg/flytekit

Pending review on flyteorg/flytekit-java - Manage reminder [#6] Update .env.local in README (kanterov) a year stale · 3 years old · Waiting on _ @Ketan (kumare3)_

#224 [Snyk] Upgrade info.picocli:picocli from 4.7.1 to 4.7.2 Pull request opened by snyk-bot Snyk has created this PR to upgrade info.picocli:picocli from 4.7.1 to 4.7.2. ℹ️ Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. * * * • The recommended version is 1 version ahead of your current version. • The recommended version was released 22 days ago, on 2023-04-09. * * * Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs. For more information: https://camo.githubusercontent.com/642b96ab7f84300036f39fc3423b3b3c84f4083cfed45a0cc61153c83c977151/68747470733a2f2f6170692e7365676d656e742e696f2f76312f706978656c2f747261636b3f646174613d65794a33636d6c305a55746c65534936496e4a79576d785a634564485932527954485a7362306c596430645563566734576b4652546e4e434f5545774969776959573576626e6c746233567a535751694f69497a4e7a64695a6d45775a6930354d5467774c54526b4f446774595455774f4330314d6d4a6a4e54466d4f5463304e444d694c434a6c646d567564434936496c425349485a705a58646c5a434973496e42796233426c636e52705a584d694f6e736963484a4a5a434936496a4d334e324a6d5954426d4c546b784f4441744e4751344f4331684e5441344c545579596d4d314d5759354e7a51304d794a3966513d3d 🧐 View latest project report 🛠️ Adjust upgrade PR settings 🔕 Ignore this dependency or unsubscribe from future upgrade PRs flyteorg/flytekit-java GitHub Actions: build-snapshot DCO: DCO ✅ 1 other check has passed 1/3 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by eapolinario

<https://github.com/flyteorg/flyteidl/commit/edc3faa12d0ce683bde19c15295ec81c34639e41|edc3faa1>

- Retract 1.4.x (#397) flyteorg/flyteidl

Release - v1.3.20 New release published by flyte-bot Changelog • edc3faa Retract 1.4.x (#397) flyteorg/flyteidl

#3632 [BUG] FlyteDirectory not copied to ContainerTask inputs Issue created by pryce-turner Describe the bug When using a ContainerTask with a FlyteDirectory in the inputs, no inputs are actually copied to the running container. For example:

bt = ContainerTask(
    name="basic-test",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(indir=FlyteDirectory),
    outputs=kwtypes(),
    image="<http://ghcr.io/flyteorg/rawcontainers-shell:v2|ghcr.io/flyteorg/rawcontainers-shell:v2>",
    command=[
        "ls",
        "-la",
        "/var/inputs",
    ],
)

@task
def get_dir(dirpath: str) -> FlyteDirectory:
    fd = FlyteDirectory(path=dirpath)
    return fd

@workflow
def wf():
    fd = get_dir(dirpath='<s3://my-s3-bucket/cv-in>')
    bt(indir=fd)

This workflow returns an empty directory at

/var/inputs

according to the k8s logs despite there definitely being objects at the specified path. Expected behavior I would expect the above workflow to list either

/var/inputs/indir

with objects copied to indir or perhaps copy them directly i.e.

/var/inputs/obj1

,

/var/inputs/obj2

as they are named in the object store. Additional context to reproduce This was run on a local sandbox via

pyflyte run --remote basic_test.py wf

Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#750 chore: guard against /tasks failing Pull request opened by ursucarina TL;DR guard against /tasks failing Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Tracking Issue NA flyteorg/flyteconsole GitHub Actions: Build & Push FlyteConsole Image GitHub Actions: Get Release Tag GitHub Actions: Generate Release Codecov: 66.53% (-0.11%) compared to 8f8d686 Codecov: 50.00% of diff hit (target 66.63%) ✅ 6 other checks have passed 6/11 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flytekit/tree/master|master>

by pingsutw

<https://github.com/flyteorg/flytekit/commit/e5b05110325a71ca229440da283e989bf1b228c6|e5b05110>

- Measuring the time taken for each component when executing a task (#1581) flyteorg/flytekit

#31 verification was failing in GPU mode [mnist-training] Pull request opened by zeryx Verification step failed when

gpu_enabled

was set to True, fixed by adding an additional conditional to dataset. flyteorg/flytekit-python-template ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyteidl/commit/513ff8221c68e3faccbdbd2039cee4b4d4e6ccf0|513ff822>

- Data addresses #minor (#391) flyteorg/flyteidl

Release - v1.4.0 New release published by flyte-bot Changelog • 513ff82 Data addresses #minor (#391) flyteorg/flyteidl

#399 #minor - bump to 1.5.x to avoid 1.4.0 and 1.4.2 issues Pull request opened by wild-endeavor Read then delete this section - Make sure to use a concise title for the pull-request. _- Use #patch, #minor or #major in the pull-request title to bump the corresponding version. Otherwise, the patch version will be bumped. More details_ TL;DR Please replace this text with a description of what this PR accomplishes. Type ☐ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☐ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue _Remove the '_fixes_' keyword if there will be multiple PRs to fix the linked issue_ fixes https://github.com/flyteorg/flyte/issues/ Follow-up issue NA OR https://github.com/flyteorg/flyte/issues/ flyteorg/flyteidl ✅ All checks have passed 2/2 successful checks

1 new commit pushed to

<https://github.com/flyteorg/flyteidl/tree/master|master>

by wild-endeavor

<https://github.com/flyteorg/flyteidl/commit/9189ff290804ae25775daf144a7c0e6511d27e15|9189ff29>

- #minor - bump to 1.5.x to avoid 1.4.0 and 1.4.2 issues (#399) flyteorg/flyteidl

Release - v1.5.0 New release published by flyte-bot Changelog • 9189ff2 #minor - bump to 1.5.x to avoid 1.4.0 and 1.4.2 issues (#399) flyteorg/flyteidl

#546 Address resolution Pull request opened by wild-endeavor TL;DR Add logic to return and accept

flyte://

when getting data Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added ☐ Any pending items have an associated Issue Complete description How did you fix the bug, make the feature etc. Link to any design docs etc Tracking Issue Related to: flyteorg/flyteidl#391 flyteorg/flyteadmin GitHub Actions: Unit Tests / Run Unit Test GitHub Actions: Docker Build Images / Build Docker Image GitHub Actions: Lint / Run Lint GitHub Actions: Check Go Generate / Go Generate ✅ 2 other checks have passed 2/6 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/4e8c868796a3a848ea2a42ec07715d831de068a0|4e8c8687>

- Update privileges, scope and nomination processes (#5) flyteorg/community

#208 Asynchronous / manual / human-in-the-loop tasks Issue created by MrSaints Motivation: Why do you think this is important? There are several data / AI / ML workflows which may require asynchronous completion. That is, rather than having the task automatically scheduled on Kubernetes or Spark, it can be asynchronously picked up, and completed. The intended audience for these sort of tasks are humans. Data / AI / ML workflows often require some "human-in-the-loop" (HITL) to audit, review, feedback, and/or adjust the results / outcomes of automated tasks. Examples: • An ETL workflow where a snapshot of the results of an extraction, and/or transformation are assessed before being loaded to avoid polluting the data target • A decision-making AI / ML workflow (e.g. risk assessment, fraud detection, etc) where less confident scores are escalated to a human operator to make the final decision • Tasks which require completion from robotic process automation (RPA) or some external / on-premise service Goal: What should the final outcome look like, ideally? Ideally, it should not deviate too much from Flyte's DSL. It may be achievable with a new type of task. In Flytekit, this may be an

@asynchronous_task

decorator. External services can pull pending tasks, and complete them through an API. Describe alternatives you've considered When discussed on Slack, Ketan recommended:

There is no plugin that waits for manual input today, though conceptually we can definitely add one.
But this can be easily achieved in Flyte without any more code. This is how you could do it
Lets says there is a workflow with task1, task2, task3, task4, task5. In this workflow task3 is actually needing a human input

Workflow1 with task1 & task2

Workflow2 with task4 & B task5

and you can now wait for Workflow1 to finish, do your logic in your own service/component etc and then call flyteAdmin api to trigger workflow2

Though this works, it relies on the external service to make the appropriate callback. That external service may not be concerned about what happens next, and baking that logic may make the whole workflow brittle. Flyte component ☑︎ Overall ☐ Flyte Setup and Installation scripts ☑︎ Flyte Documentation ☐ Flyte communication (slack/email etc) ☐ FlytePropeller ☐ FlyteIDL (Flyte specification language) ☑︎ #2631 ☐ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☐ FlyteConsole (UI) ☐ Other [Optional] Propose: Link/Inline Still thinking about this as I'm quite new to Flyte. Additional context I'm borrowing the language from quite a few other orchestrators / workflow engines with a similar feature / concept: • Uber's Cadence Workflow: https://cadenceworkflow.io/docs/07_goclient/12_activity_async_completion • Camunda's Zeebe: https://zeebe.io/blog/2018/11/message-correlation-working-with-messages-in-zeebe/ • Netflix's Conductor: https://netflix.github.io/conductor/configuration/systask/#wait • Orchestra: https://github.com/b12io/orchestra (human-in-the-loop orchestration) Is this a blocker for you to adopt Flyte Not really. The alternative provided by Ketan may work, but again, it would be better to have the visibility, and control in one place. Otherwise, it'd defeat the purpose of having a workflow engine / orchestrator managing the control flow while services can focus purely on the business logic. flyteorg/flyte

#3633 [Core feature] Improve error message on promises Issue created by cosmicBboy Motivation: Why do you think this is important? The purpose of this issue is to teach new users about how Promises work via better error messages. For example, if a user tries to do

promise.get()

or something like that, we should let them know what the legal operations are of promises (e.g. passing to other tasks/workflows, conditionals, etc.) Goal: What should the final outcome look like, ideally? TBD Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#32 Zeryx/integration development Pull request opened by zeryx This PR adds integration.py, a integration testing suite that registers and executes all workflows in templates found in flytekit-python-template; used for end-to-end integration testing. This contains a github actions workflow.yaml that triggers on new PR commit, and incrementally builds images (and deploys them to ghcr.io) if necessary. It then runs the default workflow as an execution; and if successful continues to the next. If all succeed, then the images will be removed from ghcr.io, and the action will return true for this PR; ready to be merged. flyteorg/flytekit-python-template GitHub Actions: integration_tests GitHub Actions: build_images ✅ 2 other checks have passed 2/4 successful checks

#1614 Skip the image building process if the check for its existence fails Pull request opened by pingsutw Recreate PR because #1609 stuck at pending. TL;DR Skip building the image if failed to check if image exists, and we assume image already exists. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☐ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description ^^^ Tracking Issue NA Follow-up issue NA flyteorg/flytekit Codecov: 70.10% (-0.06%) compared to e5b0511 Codecov: 29.62% of diff hit (target 70.16%) ✅ 28 other checks have passed 28/30 successful checks

#3634 [Core feature] raise warning when using default args in tasks that are not None Issue created by cosmicBboy Motivation: Why do you think this is important? Currently, flyte only supports default arguments in tasks that are

None

. We need to raise a warning if the user specifies any other value so that this feature isn't misused. Goal: What should the final outcome look like, ideally? When a user does something like:

@task
def t1(
   arg: Optional[str] = "",
)

Flytekit should raise a warning, something like:

Only None can be used as a default argument in flyte tasks

Describe alternatives you've considered NA Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte

#3631 feat: allow seedProjects to be set via inline configuration Pull request opened by ossareh Tracking issue Closes #3545 Describe your changes Add a

SeedProjects

key to the Admin configuration struct in

cmd/single

. This allows users to specify which projects should be created at start up. This allows management of the flyte installation to be more declarative via the helm chart installation path. Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☑︎ All new and existing tests passed. ☑︎ All commits are signed-off. Note to reviewers The original ticket suggests that

configuration.admin.seedProjects

should be settable; however, that key doesn't exist in the current setup. Instead this works in the same way one can set task_resources, via `configuration.inline`:

# values.yaml
flyte-core-components:
  admin:
    seedProjects:
      - flytesnacks
      - flytemeals

In my local sandbox:

POD_NAMESPACE=flyte go run -tags console cmd/main.go start --config ~/.flyte/local-dev-config.yaml

with a

~/.flyte/local-dev-config.yaml

containing:

flyte:
  admin:
    seedProjects:
      - flytesnacks
      - flytemeals

Results in: I haven't run this in a cluster; I'm not entirely sure how to do so easily. I see some infrastructure for k3s (which I've not used), but I cannot find docs on how to use that to validate this change. Reading deployment.yaml leads me to believe this should work with the proposed values.yaml above. flyteorg/flyte GitHub Actions: build-and-push-sandbox-bundled-image GitHub Actions: sandbox-bundled-functional-tests GitHub Actions: build-and-push-single-binary-image ✅ 9 other checks have passed 9/12 successful checks

#1567 Address resolution Pull request opened by wild-endeavor TL;DR This implements in FlyteRemote a quick feature to help the fetching of inputs and outputs and decks from a

flyte://

uri. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☑︎ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description Other PRs: flyteorg/flyteadmin#546 flyteorg/flyteidl#391 Tracking Issue No issues, this is just a use-ability enhancement we found flyteorg/flytekit Codecov: 47.88% of diff hit (target 70.16%) Codecov: 69.93% (-0.23%) compared to e5b0511 GitHub Actions: docs GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-snowflake) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-kf-tensorflow) GitHub Actions: build-plugins (3.11, flytekit-kf-pytorch) GitHub Actions: build-plugins (3.11, flytekit-kf-mpi) GitHub Actions: build-plugins (3.11, flytekit-k8s-pod) GitHub Actions: build-plugins (3.11, flytekit-hive) GitHub Actions: build-plugins (3.11, flytekit-envd) GitHub Actions: build-plugins (3.11, flytekit-duckdb) GitHub Actions: build-plugins (3.11, flytekit-dolt) GitHub Actions: build-plugins (3.11, flytekit-deck-standard) GitHub Actions: build-plugins (3.11, flytekit-dbt) GitHub Actions: build-plugins (3.11, flytekit-data-fsspec) GitHub Actions: build-plugins (3.11, flytekit-dask) GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.10, flytekit-vaex) ✅ 8 other checks have passed 8/30 successful checks

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/418628c31fc112bbd92fe808c1ac505ff03cc8d7|418628c3>

- Add adopters meetup notes placeholder flyteorg/community

1 new commit pushed to

<https://github.com/flyteorg/community/tree/main|main>

by davidmirror-ops

<https://github.com/flyteorg/community/commit/9075a4bde832d48e52d01e05f07898d4832f311c|9075a4bd>

- May 2nd meeting notes flyteorg/community