flyte-org #announcements

Madhur Tandon

07/03/2022, 5:19 AM

I guess all issues are opened in the “flyte” repository, even for other repositories.

George Odette

07/03/2022, 10:41 AM

Untitled.txt

George Odette

07/03/2022, 10:41 AM

HI, I am running a workflow and getting the error Below. I cannot seem to understand what it needs since any help?

George Odette

07/05/2022, 10:43 AM

Hey @Haytham Abuelfutuh let me know of your next office hours for some consultation

seunggs

07/05/2022, 4:52 PM

Hi, can anyone point to some documentation on cluster_resource_manager (looking at the helm chart values)?

seunggs

07/05/2022, 5:00 PM

Another question - my helm chart doesn’t seem to be applying values-eks.yaml

seunggs

07/05/2022, 5:00 PM

When I run

kubectl describe cm flyte-admin-base-config

, I see these values

seunggs

07/05/2022, 5:01 PM

Copy code

db.yaml:
----
database:
  dbname: flyteadmin
  host: postgres
  port: 5432
  username: postgres

seunggs

07/05/2022, 5:01 PM

It’s not using my db host, etc. And some google account I don’t recognize:

seunggs

07/05/2022, 5:01 PM

Copy code

server.yaml:
----
auth:
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - offline
        - all
  authorizedUris:
  - <https://localhost:30081>
  - <http://flyteadmin:80>
  - <http://flyteadmin.flyte.svc.cluster.local:80>
  userAuth:
    openId:
      baseUrl: <https://accounts.google.com>
      clientId: <http://657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com|657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com>
      scopes:
      - profile
      - openid

seunggs

07/05/2022, 5:01 PM

Is this expected?

seunggs

07/05/2022, 5:03 PM

This might be the reason why I’m getting

<https://flyte.sidetrek.com/me> (501)

not implemented error in the flyte console

seunggs

07/05/2022, 5:03 PM

"unknown service flyteidl.service.IdentityService"

krishna Yerramsetty

07/05/2022, 5:40 PM

Hello all! Question around the right way of using dynamic workflows with mapped tasks inside them. I adapted the map_task_example to run within a dynamic workflow instead of the general workflow like so:

Copy code

@dynamic
def wrapper_task(a: typing.List[int]) -> str:
    mapped_out = map_task(a_mappable_task)(a=a).with_overrides(
        retries=4,
    )
    coalesced = coalesce(b=mapped_out)
    return coalesced


@task
def a_mappable_task(a: int) -> str:
    inc = a + 2
    stringified = str(inc)
    return stringified


@task
def coalesce(b: typing.List[str]) -> str:
    coalesced = "".join(b)
    return coalesced


@workflow
def my_map_workflow(a: typing.List[int]) -> str:

    return wrapper_task(a=a)

    if __name__ == "__main__":
        result = my_map_workflow(a=[1, 2, 3, 4, 5])
        print(f"{result}")

I am seeing a type mismatch compilation error however at run-time:

Copy code

User] malformed dynamic workflow, caused by: Collected Errors: 2
	Error 0: Code: MismatchingTypes, Node Id: dn1, Description: Variable [o0] (type [collection_type:<collection_type:<simple:STRING > > ]) doesn't match expected type [collection_type:<simple:STRING > ].

Any ideas on why the return is a list of lists from the mapped task? And how do I correctly pass these outputs onto other tasks?

Sandra Youssef

07/05/2022, 7:43 PM

Hi Flyers, If you'd like to learn more about one of Flyte's Ecosystem Projects, UnionML, join @Niels Bantilan in our very first UnionML OSS Planning event! • Bring your questions, ideas and feedback • Take part in the UnionML roadmap discussion Open to the whole community! Tomorrow Wednesday July 6 at 9am PT. Calendar Invite See you there! Flyte Team

Vladimir Lialine

07/06/2022, 12:24 AM

Hello Flyte users.

🖖 2

👋 7

Bernhard Stadlbauer

07/06/2022, 7:09 AM

Hi! We've installed the

flyte-core

helm

chart in version

1.0.2

on one of our clusters yesterday, and it came up successfully. Whilst trying to upgrade another cluster today, we did get an an

ImagePullBackoff

when trying to pull

<http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>

. I can also confirm not being able to pull that image locally:

Copy code

❯ docker pull <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>
Error response from daemon: manifest unknown

Has anything changed here recently?

Klemens Kasseroller

07/06/2022, 1:28 PM

Hi! While trying to launch workflows from the flyte console, I experienced timeouts with the error message "Request failed with status code 504" coming from the flyte admin after 15s. I figured out, that the problem occurs if the inputs and outputs of the individual tasks are complex/nested dataclasses. Could someone explain to me, what is happening behind the scenes, when launching a (static) workflow, and why the structure of my inputs and outputs matters at that point in time. Does anyone know a solution to this problem? I am using the sandbox deployment (not the DinD deployment) on a local Kubernetes cluster (tested both on a Windows laptop and on a Linux workstation). Chart versions:

flyte-core-v1.0.2

flyte-deps-v1.0.2

Sonja Ericsson

07/06/2022, 1:37 PM

hey! is there a metric in propeller to measure failures from plugins? We use these recording rules currently, but each time we add a new plugin we need to go and add it to the rules 🧵

Mike Zhong

07/06/2022, 1:51 PM

Hello flyte team, I opened an issue about a week ago regarding what I believe to be a bug with the flyte console UI. If someone could give it a quick triage, that would be great. Thanks https://github.com/flyteorg/flyte/issues/2637

👀 1

Matheus Moreno

07/06/2022, 2:23 PM

Hi, everyone! Two people in my team are stuck on the same problem. When trying to start the sandbox, this error keeps appearing:

Copy code

Deploying Flyte-deps...
Release "flyte-deps" does not exist. Installing it now.
Error: failed to download "flyteorg/flyte-deps" at version "v1.1.0" (hint: running helm repo update may help)

We already tried removing the

.kube

.cache

and

.local

directories, and changing the sandbox version. I have no idea what else we can do.

Haytham Abuelfutuh

07/06/2022, 5:46 PM

<!here>, we have included an important security fix in the flyte v1.1.0 release. We highly encourage everyone to upgrade. A CVE will be released on July 12th that will include more information about the vulnerability. If you have specific questions, please don’t hesitate to DM me until then.

👍 4

austin

07/06/2022, 10:31 PM

Submitted a PR to a repo in unionai-oss github [ specifically in https://github.com/unionai-oss/flyte-register-action ] … What’s the workflow for people to update those repos ( since isn’t as closely tied to the flyte ecosystem … do we not consider that as much of a “community” environment )? Related, there didn’t appear to be a way to tag/assign a reviewer…. LOL, this might not even be the right place to ask that question — but I don’t know where else would be 😛

seunggs

07/07/2022, 4:00 AM

Hi, when I deploy Flyte in EKS via Helm chart (as per the manual eks deployment doc), I see these errors in the console - is this to be expected? (I’m using the default helm values)

seunggs

07/07/2022, 4:00 AM

message has been deleted

seunggs

07/07/2022, 4:01 AM

This is in the route:

/console/projects/flyteexamples/executions?duration=all

Sandra Youssef

07/07/2022, 3:21 PM

Hi Flyers, Meet the Flyte-Airflow Provider, a package able to extend Airflow to accommodate MLOps requirements. By Union.ai Software Engineer & Tech Evangelist, @Samhita Alla https://github.com/flyteorg/airflow-provider-flyte/tree/master

👍 2

🙌 4

😎 2

🦜 2

Sandra Youssef

07/07/2022, 10:37 PM

Hi Flyers, Meet the contributors of UnionML, find out about upcoming milestones, how to file issues, and the preliminary roadmap of UnionML, from the very first UnionML OSS Planning event with @Niels Bantilan. Meet us again on 7/20 at 9am PT!

👍 1

Evan Sadler

07/08/2022, 2:25 PM

I have an offline model pipeline that works 100% in spark: preprocessing, training LightGBM (synapse ml), and predictions. I am curious if this could fit into the UnionML framework. This would allow all the jobs to scale really nicely. I can see moving the preprocessing into the Dataset and then swapping out the DataFrame for a spark dataframe. There would need to be some config to have the tasks create spark jobs. Anyways let me know if this doesn’t sound crazy haha

Evan Sadler

07/11/2022, 4:15 PM

Hello! I am trying to use the Flyte Spark plugin with the SynapseML package with the local runner. It works until I try to add the

spark.jars.packages

to the config (see example below). There is some kind of connection refused error:

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:

. I imagine this has to do with some configuration that I don’t quite understand in the backend. Any help is much appreciated. I tested it out with another package and it had the same error.

Copy code

import datetime
import random
from operator import add

import flytekit
from flytekit import Resources, task, workflow


from flytekitplugins.spark import Spark

@task(
    task_config=Spark(
        # this configuration is applied to the spark cluster
        spark_conf={
            "spark.driver.memory": "8g",
            "spark.jars.repositories": "<https://mmlspark.azureedge.net/maven>", 
            "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5"  # adding this causes problems
        }
    ),
    limits=Resources(mem="2000M"),
    cache_version="1",
)
def hello_spark(partitions: int) -> float:
    print("Starting Spark with Partitions: {}".format(partitions))

    n = 100000 * partitions
    sess = flytekit.current_context().spark_session
    count = (
        sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
    )
    pi_val = 4.0 * count / n
    print("Pi val is :{}".format(pi_val))
    return pi_val