flyte-org #announcements

announcements

Sugato Ray

07/02/2022, 10:47 PM

I wanted to open an issue for

flytekit

. But there is no option for opening an issue at the GitHub repository! Seems like you can only open pull requests. Am I missing something? • https://github.com/flyteorg/flytekit

Madhur Tandon

07/03/2022, 5:19 AM

I guess all issues are opened in the “flyte” repository, even for other repositories.

George Odette

07/03/2022, 10:41 AM

Untitled.txt

George Odette

07/03/2022, 10:41 AM

HI, I am running a workflow and getting the error Below. I cannot seem to understand what it needs since any help?

George Odette

07/05/2022, 10:43 AM

Hey @Haytham Abuelfutuh let me know of your next office hours for some consultation

seunggs

07/05/2022, 4:52 PM

Hi, can anyone point to some documentation on cluster_resource_manager (looking at the helm chart values)?

seunggs

07/05/2022, 5:00 PM

Another question - my helm chart doesn’t seem to be applying values-eks.yaml

seunggs

07/05/2022, 5:00 PM

When I run

kubectl describe cm flyte-admin-base-config

, I see these values

seunggs

07/05/2022, 5:01 PM

db.yaml:
----
database:
  dbname: flyteadmin
  host: postgres
  port: 5432
  username: postgres

seunggs

07/05/2022, 5:01 PM

It’s not using my db host, etc. And some google account I don’t recognize:

seunggs

07/05/2022, 5:01 PM

server.yaml:
----
auth:
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - offline
        - all
  authorizedUris:
  - <https://localhost:30081>
  - <http://flyteadmin:80>
  - <http://flyteadmin.flyte.svc.cluster.local:80>
  userAuth:
    openId:
      baseUrl: <https://accounts.google.com>
      clientId: <http://657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com|657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com>
      scopes:
      - profile
      - openid

seunggs

07/05/2022, 5:01 PM

Is this expected?

seunggs

07/05/2022, 5:03 PM

This might be the reason why I’m getting

<https://flyte.sidetrek.com/me> (501)

not implemented error in the flyte console

seunggs

07/05/2022, 5:03 PM

"unknown service flyteidl.service.IdentityService"

krishna Yerramsetty

07/05/2022, 5:40 PM

Hello all! Question around the right way of using dynamic workflows with mapped tasks inside them. I adapted the map_task_example to run within a dynamic workflow instead of the general workflow like so:

@dynamic
def wrapper_task(a: typing.List[int]) -> str:
    mapped_out = map_task(a_mappable_task)(a=a).with_overrides(
        retries=4,
    )
    coalesced = coalesce(b=mapped_out)
    return coalesced


@task
def a_mappable_task(a: int) -> str:
    inc = a + 2
    stringified = str(inc)
    return stringified


@task
def coalesce(b: typing.List[str]) -> str:
    coalesced = "".join(b)
    return coalesced


@workflow
def my_map_workflow(a: typing.List[int]) -> str:

    return wrapper_task(a=a)

    if __name__ == "__main__":
        result = my_map_workflow(a=[1, 2, 3, 4, 5])
        print(f"{result}")

I am seeing a type mismatch compilation error however at run-time:

User] malformed dynamic workflow, caused by: Collected Errors: 2
	Error 0: Code: MismatchingTypes, Node Id: dn1, Description: Variable [o0] (type [collection_type:<collection_type:<simple:STRING > > ]) doesn't match expected type [collection_type:<simple:STRING > ].

Any ideas on why the return is a list of lists from the mapped task? And how do I correctly pass these outputs onto other tasks?

Sandra Youssef

07/05/2022, 7:43 PM

Hi Flyers, If you'd like to learn more about one of Flyte's Ecosystem Projects, UnionML, join @Niels Bantilan in our very first UnionML OSS Planning event! • Bring your questions, ideas and feedback • Take part in the UnionML roadmap discussion Open to the whole community! Tomorrow Wednesday July 6 at 9am PT. Calendar Invite See you there! Flyte Team

Vladimir Lialine

07/06/2022, 12:24 AM

Hello Flyte users.

🖖 2

👋 7

Bernhard Stadlbauer

07/06/2022, 7:09 AM

Hi! We've installed the

flyte-core

helm

chart in version

1.0.2

on one of our clusters yesterday, and it came up successfully. Whilst trying to upgrade another cluster today, we did get an an

ImagePullBackoff

when trying to pull

<http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>

. I can also confirm not being able to pull that image locally:

❯ docker pull <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>
Error response from daemon: manifest unknown

Has anything changed here recently?

Klemens Kasseroller

07/06/2022, 1:28 PM

Hi! While trying to launch workflows from the flyte console, I experienced timeouts with the error message "Request failed with status code 504" coming from the flyte admin after 15s. I figured out, that the problem occurs if the inputs and outputs of the individual tasks are complex/nested dataclasses. Could someone explain to me, what is happening behind the scenes, when launching a (static) workflow, and why the structure of my inputs and outputs matters at that point in time. Does anyone know a solution to this problem? I am using the sandbox deployment (not the DinD deployment) on a local Kubernetes cluster (tested both on a Windows laptop and on a Linux workstation). Chart versions:

flyte-core-v1.0.2

flyte-deps-v1.0.2

Sonja Ericsson

07/06/2022, 1:37 PM

hey! is there a metric in propeller to measure failures from plugins? We use these recording rules currently, but each time we add a new plugin we need to go and add it to the rules 🧵

Mike Zhong

07/06/2022, 1:51 PM

Hello flyte team, I opened an issue about a week ago regarding what I believe to be a bug with the flyte console UI. If someone could give it a quick triage, that would be great. Thanks https://github.com/flyteorg/flyte/issues/2637

👀 1

Matheus Moreno

07/06/2022, 2:23 PM

Hi, everyone! Two people in my team are stuck on the same problem. When trying to start the sandbox, this error keeps appearing:

Deploying Flyte-deps...
Release "flyte-deps" does not exist. Installing it now.
Error: failed to download "flyteorg/flyte-deps" at version "v1.1.0" (hint: running helm repo update may help)

We already tried removing the

.kube

.cache

and

.local

directories, and changing the sandbox version. I have no idea what else we can do.

Haytham Abuelfutuh

07/06/2022, 5:46 PM

<!here>, we have included an important security fix in the flyte v1.1.0 release. We highly encourage everyone to upgrade. A CVE will be released on July 12th that will include more information about the vulnerability. If you have specific questions, please don’t hesitate to DM me until then.

👍 4

austin

07/06/2022, 10:31 PM

Submitted a PR to a repo in unionai-oss github [ specifically in https://github.com/unionai-oss/flyte-register-action ] … What’s the workflow for people to update those repos ( since isn’t as closely tied to the flyte ecosystem … do we not consider that as much of a “community” environment )? Related, there didn’t appear to be a way to tag/assign a reviewer…. LOL, this might not even be the right place to ask that question — but I don’t know where else would be 😛

seunggs

07/07/2022, 4:00 AM

Hi, when I deploy Flyte in EKS via Helm chart (as per the manual eks deployment doc), I see these errors in the console - is this to be expected? (I’m using the default helm values)

seunggs

07/07/2022, 4:00 AM

seunggs

07/07/2022, 4:01 AM

This is in the route:

/console/projects/flyteexamples/executions?duration=all

Sandra Youssef

07/07/2022, 3:21 PM

Hi Flyers, Meet the Flyte-Airflow Provider, a package able to extend Airflow to accommodate MLOps requirements. By Union.ai Software Engineer & Tech Evangelist, @Samhita Alla https://github.com/flyteorg/airflow-provider-flyte/tree/master

👍 2

🙌 4

😎 2

😛artyparrot: 2

Sandra Youssef

07/07/2022, 10:37 PM

Hi Flyers, Meet the contributors of UnionML, find out about upcoming milestones, how to file issues, and the preliminary roadmap of UnionML, from the very first UnionML OSS Planning event with @Niels Bantilan. Meet us again on 7/20 at 9am PT!

👍 1

Evan Sadler

07/08/2022, 2:25 PM

I have an offline model pipeline that works 100% in spark: preprocessing, training LightGBM (synapse ml), and predictions. I am curious if this could fit into the UnionML framework. This would allow all the jobs to scale really nicely. I can see moving the preprocessing into the Dataset and then swapping out the DataFrame for a spark dataframe. There would need to be some config to have the tasks create spark jobs. Anyways let me know if this doesn’t sound crazy haha

Title

Evan Sadler

07/08/2022, 2:25 PM

Niels Bantilan

07/08/2022, 2:48 PM

This is not crazy at all! Need to improve the docs around this, but you can pass in all the

@task

kwargs to the following decorators: • Dataset.reader • Model.trainer • Model.predictor Meaning you can specify

SparkConfig

or any other Flyte-compatible task type configuration that typically work with Flyte tasks.

🙏 2

Evan Sadler

07/08/2022, 2:53 PM

🤯 🤯 🤯

Alright I am going to use it next week 🙂

😛artyparrot: 1

@Niels Bantilan say I want to run a model a few different times with different features sets and possibly filtering rows (like subset to people in the USA). • For running variations, I believe I can just wrap these in flyte tasks • I see how to change the features, but not how to subset rows through the Dataset class. Do you have any recommendations for the second one?

Niels Bantilan

07/08/2022, 4:25 PM

You can vary the features. Dataset.reader The dataset reader should output your dataset, and you can parameterize this function however way you wish. Here’s a basic example

@dataset.reader
def reader(feature_set: str, row_filter: str) -> pd.DataFrame
    data = ...  # get data using a SQL query, or whatever
    if feature_set == "feature_set_1":
        selected_data = ...
    elif feature_set == "feature_set_2":
        selected_data = ...
    else:
        ...  # etc
    
    if row_filter == "something":
        filtered_data = ...
    elif: ...

    return selected_data

🙏 1

🎉 1

You can then pass these parameters to model.train, where the

***reader_kwargs*

are forwarded to the reader function.

Evan Sadler

07/08/2022, 4:27 PM

So helpful!

Niels Bantilan

07/08/2022, 4:27 PM

Actually, would you mind creating an issue to document this use case? https://github.com/unionai-oss/unionml/issues/new Would love to capture it and add a page about this in the docs

Evan Sadler

07/08/2022, 4:28 PM

Yeah of course!

👍 1

Niels Bantilan

07/08/2022, 4:32 PM

Come to think of it, the Dataset.loader decorator might be a better place to vary datasets based on per-run parameters, but currently this isn’t supported. Working on this issue to address this tho

Evan Sadler

07/08/2022, 5:20 PM

I added an issue and yes I agree to the dataset loader being a great place.

View count: 2