https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • s

    Sugato Ray

    07/02/2022, 10:47 PM
    I wanted to open an issue for
    flytekit
    . But there is no option for opening an issue at the GitHub repository! Seems like you can only open pull requests. Am I missing something? • https://github.com/flyteorg/flytekit
    n
    • 2
    • 2
  • m

    Madhur Tandon

    07/03/2022, 5:19 AM
    I guess all issues are opened in the “flyte” repository, even for other repositories.
  • g

    George Odette

    07/03/2022, 10:41 AM
    Untitled.txt
  • g

    George Odette

    07/03/2022, 10:41 AM
    HI, I am running a workflow and getting the error Below. I cannot seem to understand what it needs since any help?
    k
    • 2
    • 4
  • g

    George Odette

    07/05/2022, 10:43 AM
    Hey @Haytham Abuelfutuh let me know of your next office hours for some consultation
    s
    • 2
    • 3
  • s

    seunggs

    07/05/2022, 4:52 PM
    Hi, can anyone point to some documentation on cluster_resource_manager (looking at the helm chart values)?
    k
    • 2
    • 4
  • s

    seunggs

    07/05/2022, 5:00 PM
    Another question - my helm chart doesn’t seem to be applying values-eks.yaml
  • s

    seunggs

    07/05/2022, 5:00 PM
    When I run
    kubectl describe cm flyte-admin-base-config
    , I see these values
  • s

    seunggs

    07/05/2022, 5:01 PM
    db.yaml:
    ----
    database:
      dbname: flyteadmin
      host: postgres
      port: 5432
      username: postgres
  • s

    seunggs

    07/05/2022, 5:01 PM
    It’s not using my db host, etc. And some google account I don’t recognize:
  • s

    seunggs

    07/05/2022, 5:01 PM
    server.yaml:
    ----
    auth:
      appAuth:
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: <http://localhost:53593/callback>
            scopes:
            - offline
            - all
      authorizedUris:
      - <https://localhost:30081>
      - <http://flyteadmin:80>
      - <http://flyteadmin.flyte.svc.cluster.local:80>
      userAuth:
        openId:
          baseUrl: <https://accounts.google.com>
          clientId: <http://657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com|657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com>
          scopes:
          - profile
          - openid
  • s

    seunggs

    07/05/2022, 5:01 PM
    Is this expected?
  • s

    seunggs

    07/05/2022, 5:03 PM
    This might be the reason why I’m getting
    <https://flyte.sidetrek.com/me> (501)
    not implemented error in the flyte console
  • s

    seunggs

    07/05/2022, 5:03 PM
    "unknown service flyteidl.service.IdentityService"
  • k

    krishna Yerramsetty

    07/05/2022, 5:40 PM
    Hello all! Question around the right way of using dynamic workflows with mapped tasks inside them. I adapted the map_task_example to run within a dynamic workflow instead of the general workflow like so:
    @dynamic
    def wrapper_task(a: typing.List[int]) -> str:
        mapped_out = map_task(a_mappable_task)(a=a).with_overrides(
            retries=4,
        )
        coalesced = coalesce(b=mapped_out)
        return coalesced
    
    
    @task
    def a_mappable_task(a: int) -> str:
        inc = a + 2
        stringified = str(inc)
        return stringified
    
    
    @task
    def coalesce(b: typing.List[str]) -> str:
        coalesced = "".join(b)
        return coalesced
    
    
    @workflow
    def my_map_workflow(a: typing.List[int]) -> str:
    
        return wrapper_task(a=a)
    
        if __name__ == "__main__":
            result = my_map_workflow(a=[1, 2, 3, 4, 5])
            print(f"{result}")
    I am seeing a type mismatch compilation error however at run-time:
    User] malformed dynamic workflow, caused by: Collected Errors: 2
    	Error 0: Code: MismatchingTypes, Node Id: dn1, Description: Variable [o0] (type [collection_type:<collection_type:<simple:STRING > > ]) doesn't match expected type [collection_type:<simple:STRING > ].
    Any ideas on why the return is a list of lists from the mapped task? And how do I correctly pass these outputs onto other tasks?
    y
    • 2
    • 9
  • s

    Sandra Youssef

    07/05/2022, 7:43 PM
    Hi Flyers, If you'd like to learn more about one of Flyte's Ecosystem Projects, UnionML, join @Niels Bantilan in our very first UnionML OSS Planning event! • Bring your questions, ideas and feedback • Take part in the UnionML roadmap discussion Open to the whole community! Tomorrow Wednesday July 6 at 9am PT. Calendar Invite See you there! Flyte Team
  • v

    Vladimir Lialine

    07/06/2022, 12:24 AM
    Hello Flyte users.
    🖖 2
    👋 7
  • b

    Bernhard Stadlbauer

    07/06/2022, 7:09 AM
    Hi! We've installed the
    flyte-core
    helm
    chart in version
    1.0.2
    on one of our clusters yesterday, and it came up successfully. Whilst trying to upgrade another cluster today, we did get an an
    ImagePullBackoff
    when trying to pull
    <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>
    . I can also confirm not being able to pull that image locally:
    ❯ docker pull <http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>
    Error response from daemon: manifest unknown
    Has anything changed here recently?
    n
    y
    • 3
    • 2
  • k

    Klemens Kasseroller

    07/06/2022, 1:28 PM
    Hi! While trying to launch workflows from the flyte console, I experienced timeouts with the error message "Request failed with status code 504" coming from the flyte admin after 15s. I figured out, that the problem occurs if the inputs and outputs of the individual tasks are complex/nested dataclasses. Could someone explain to me, what is happening behind the scenes, when launching a (static) workflow, and why the structure of my inputs and outputs matters at that point in time. Does anyone know a solution to this problem? I am using the sandbox deployment (not the DinD deployment) on a local Kubernetes cluster (tested both on a Windows laptop and on a Linux workstation). Chart versions:
    flyte-core-v1.0.2
    flyte-deps-v1.0.2
    • 1
    • 1
  • s

    Sonja Ericsson

    07/06/2022, 1:37 PM
    hey! is there a metric in propeller to measure failures from plugins? We use these recording rules currently, but each time we add a new plugin we need to go and add it to the rules 🧵
    k
    • 2
    • 8
  • m

    Mike Zhong

    07/06/2022, 1:51 PM
    Hello flyte team, I opened an issue about a week ago regarding what I believe to be a bug with the flyte console UI. If someone could give it a quick triage, that would be great. Thanks https://github.com/flyteorg/flyte/issues/2637
    👀 1
    j
    e
    • 3
    • 3
  • m

    Matheus Moreno

    07/06/2022, 2:23 PM
    Hi, everyone! Two people in my team are stuck on the same problem. When trying to start the sandbox, this error keeps appearing:
    Deploying Flyte-deps...
    Release "flyte-deps" does not exist. Installing it now.
    Error: failed to download "flyteorg/flyte-deps" at version "v1.1.0" (hint: running helm repo update may help)
    We already tried removing the
    .kube
    ,
    .cache
    and
    .local
    directories, and changing the sandbox version. I have no idea what else we can do.
    e
    y
    +2
    • 5
    • 8
  • h

    Haytham Abuelfutuh

    07/06/2022, 5:46 PM
    <!here>, we have included an important security fix in the flyte v1.1.0 release. We highly encourage everyone to upgrade. A CVE will be released on July 12th that will include more information about the vulnerability. If you have specific questions, please don’t hesitate to DM me until then.
    👍 4
    r
    s
    y
    • 4
    • 9
  • a

    austin

    07/06/2022, 10:31 PM
    Submitted a PR to a repo in unionai-oss github [ specifically in https://github.com/unionai-oss/flyte-register-action ] … What’s the workflow for people to update those repos ( since isn’t as closely tied to the flyte ecosystem … do we not consider that as much of a “community” environment )? Related, there didn’t appear to be a way to tag/assign a reviewer…. LOL, this might not even be the right place to ask that question — but I don’t know where else would be 😛
    y
    e
    • 3
    • 3
  • s

    seunggs

    07/07/2022, 4:00 AM
    Hi, when I deploy Flyte in EKS via Helm chart (as per the manual eks deployment doc), I see these errors in the console - is this to be expected? (I’m using the default helm values)
  • s

    seunggs

    07/07/2022, 4:00 AM
    p
    • 2
    • 11
  • s

    seunggs

    07/07/2022, 4:01 AM
    This is in the route:
    /console/projects/flyteexamples/executions?duration=all
  • s

    Sandra Youssef

    07/07/2022, 3:21 PM
    Hi Flyers, Meet the Flyte-Airflow Provider, a package able to extend Airflow to accommodate MLOps requirements. By Union.ai Software Engineer & Tech Evangelist, @Samhita Alla https://github.com/flyteorg/airflow-provider-flyte/tree/master
    👍 2
    🙌 4
    😎 2
    😛artyparrot: 2
  • s

    Sandra Youssef

    07/07/2022, 10:37 PM
    Hi Flyers, Meet the contributors of UnionML, find out about upcoming milestones, how to file issues, and the preliminary roadmap of UnionML, from the very first UnionML OSS Planning event with @Niels Bantilan. Meet us again on 7/20 at 9am PT!
    👍 1
  • e

    Evan Sadler

    07/08/2022, 2:25 PM
    I have an offline model pipeline that works 100% in spark: preprocessing, training LightGBM (synapse ml), and predictions. I am curious if this could fit into the UnionML framework. This would allow all the jobs to scale really nicely. I can see moving the preprocessing into the Dataset and then swapping out the DataFrame for a spark dataframe. There would need to be some config to have the tasks create spark jobs. Anyways let me know if this doesn’t sound crazy haha
    n
    • 2
    • 11
Powered by Linen
Title
e

Evan Sadler

07/08/2022, 2:25 PM
I have an offline model pipeline that works 100% in spark: preprocessing, training LightGBM (synapse ml), and predictions. I am curious if this could fit into the UnionML framework. This would allow all the jobs to scale really nicely. I can see moving the preprocessing into the Dataset and then swapping out the DataFrame for a spark dataframe. There would need to be some config to have the tasks create spark jobs. Anyways let me know if this doesn’t sound crazy haha
n

Niels Bantilan

07/08/2022, 2:48 PM
This is not crazy at all! Need to improve the docs around this, but you can pass in all the
@task
kwargs to the following decorators: • Dataset.reader • Model.trainer • Model.predictor Meaning you can specify
SparkConfig
or any other Flyte-compatible task type configuration that typically work with Flyte tasks.
🙏 2
e

Evan Sadler

07/08/2022, 2:53 PM
🤯 🤯 🤯
Alright I am going to use it next week 🙂
😛artyparrot: 1
@Niels Bantilan say I want to run a model a few different times with different features sets and possibly filtering rows (like subset to people in the USA). • For running variations, I believe I can just wrap these in flyte tasks • I see how to change the features, but not how to subset rows through the Dataset class. Do you have any recommendations for the second one?
n

Niels Bantilan

07/08/2022, 4:25 PM
You can vary the features. Dataset.reader The dataset reader should output your dataset, and you can parameterize this function however way you wish. Here’s a basic example
@dataset.reader
def reader(feature_set: str, row_filter: str) -> pd.DataFrame
    data = ...  # get data using a SQL query, or whatever
    if feature_set == "feature_set_1":
        selected_data = ...
    elif feature_set == "feature_set_2":
        selected_data = ...
    else:
        ...  # etc
    
    if row_filter == "something":
        filtered_data = ...
    elif: ...

    return selected_data
🙏 1
🎉 1
You can then pass these parameters to model.train, where the
***reader_kwargs*
are forwarded to the reader function.
e

Evan Sadler

07/08/2022, 4:27 PM
So helpful!
n

Niels Bantilan

07/08/2022, 4:27 PM
Actually, would you mind creating an issue to document this use case? https://github.com/unionai-oss/unionml/issues/new Would love to capture it and add a page about this in the docs
e

Evan Sadler

07/08/2022, 4:28 PM
Yeah of course!
👍 1
n

Niels Bantilan

07/08/2022, 4:32 PM
Come to think of it, the Dataset.loader decorator might be a better place to vary datasets based on per-run parameters, but currently this isn’t supported. Working on this issue to address this tho
e

Evan Sadler

07/08/2022, 5:20 PM
I added an issue and yes I agree to the dataset loader being a great place.
View count: 2