https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • m

    Matheus Moreno

    04/27/2022, 6:50 PM
    Hi, everyone! Quick question: is it possible to raise the memory limit on a Flyte sandbox? A member of my team is trying to test their system locally, but is being greeted with an OOMKilled error.
    k
    • 2
    • 22
  • s

    Sandra Youssef

    04/27/2022, 7:00 PM
    Hi Flyers, If you didn't get a chance to attend our meetup with the MLOps community last week, you can now watch the just-released recording!

    https://www.youtube.com/watch?v=OLD5-G9R9fw▾

  • s

    Sam Peers Nitzberg

    04/27/2022, 7:18 PM
    Hey Flyters, bit of a self-promotion here but I produce a podcast about open-source software called Contributor, and we just featured Flyte founder Ketan Umare on today's episode: https://www.contributor.fyi/flyte
    🙌 8
    ❤️ 4
    🎉 5
  • s

    Sam Peers Nitzberg

    04/27/2022, 7:19 PM
    Thanks for checking it out!
    s
    • 2
    • 1
  • a

    Alex Bain

    04/28/2022, 12:48 AM
    @katrina there were some changes to the auth in Flyte Admin a couple weeks ago that are now in
    v1.0.0
    that have partially broken the
    kubernetes_service_account
    , i.e. some of our workflows are executing with the declared service account and some are just executing with
    default
    . I'm still working on debugging it.
    p
    k
    • 3
    • 14
  • a

    Ahmad Islam

    04/28/2022, 6:50 AM
    Hi there. I have Docker containers deployed on GKE. They got deployed and starts running at once. Output of one is based as input of next incoming container. Do we have such type of support in flyte so that it allows first one to finish and then let the next in the row to perform its job?
    s
    • 2
    • 7
  • t

    Tim Bauer

    04/28/2022, 2:42 PM
    We started working on adding a Flyte source extension to Datahub for tracking metadata and lineage about our Workflows. Is anyone else maybe already working on something similar?
    👀 2
    h
    h
    k
    • 4
    • 11
  • h

    Hafsa Junaid

    04/28/2022, 8:53 PM
    Hi What should be the output of the command (from the GCP(GKE) setup docs) which verifies the connection of db with GKE? I am getting attached outcome. For both Public and Private SQL IP. Whereas I am able to connect to psql with my local system through Public IP
    p
    • 2
    • 1
  • a

    Ahmad Islam

    04/29/2022, 5:10 AM
    Can someone share a working example for
    Multiple Image Based Implementation in Flyte
    . I can't see the implementation in docs as it is left as todo and I'm
    new to Flyte
    .
    s
    • 2
    • 2
  • s

    Sandra Youssef

    04/29/2022, 5:01 PM
    Hi Flyers, Flyte Monthly - Issue #7 is finally here! Featuring: • Flyte 1.0 Phoenix • Quickdraw Recognition demo at PyCon StartUp Row this week • The story of Flyte • MLOps Community meetup • Success Stories, and more! https://www.getrevue.co/profile/flyte/issues/flyte-monthly-issue-7-1087090 Subscribe here: https://t.co/3gycgieIgv
    :flyte: 1
    🚀 4
    🎉 5
  • a

    Ahmad Islam

    04/29/2022, 7:12 PM
    How to
    map volumes
    through flyte from a directory inside VM to a directory inside Container and vice versa.(Bind mount)
    s
    p
    • 3
    • 5
  • a

    Ahmad Islam

    05/01/2022, 6:47 PM
    Raw containers cannot be run locally at the moment.
    What does this statement mean? So, even if I have local images, I am unable to implement Flyte on them?
    k
    b
    • 3
    • 5
  • s

    Sandra Youssef

    05/02/2022, 3:52 PM
    Hi Flyers, Join us tomorrow 5/3 at the Flyte Community Sync, for updates and learn about the all-new UnionML! Calendar invite and Zoom link: https://addevent.com/event/EA7823958 See you then! Flyte Team
    🔥 5
    🚀 3
    k
    • 2
    • 1
  • a

    Alex Pozimenko

    05/02/2022, 6:17 PM
    hi team, is it possible to get recent failed executions across all projects using flyte admin api?
    k
    • 2
    • 6
  • b

    Bernhard Stadlbauer

    05/03/2022, 7:37 AM
    Hi! If I define a workflow with a dataclass/struct as input, along the lines of:
    @dataclass_json
    @dataclass
    class Foo:
        bar: int
        baz: float
        
    
    @workflow
    def my_wf(foo: Foo):
        ...
    Is it possible to introspect the fields of the
    STRUCT
    or is this checked at runtime?
    k
    • 2
    • 5
  • b

    Babis Kiosidis

    05/03/2022, 8:02 AM
    It seems that after this change our flyte services have excessive logging related to gorm, logging queries and formatting is ignored etc. more in the 🧵
    k
    • 2
    • 12
  • s

    Sören Brunk

    05/03/2022, 9:08 AM
    I just tried upgrading to
    flyteadmin:1.0.1
    for the stow fix https://github.com/flyteorg/flyteadmin/pull/416 but now flyteconsole fails with a 500. "Request failed with status code 500 header key "userinfo" contains value with non-printable ASCII characters"
    k
    p
    • 3
    • 22
  • h

    Hampus Rosvall

    05/03/2022, 1:33 PM
    Hey, looking into configuring Flyte to access GPUs. Where do I specify the plugins in the Helm values for the Flytepropeller? Do they go here, or should I enter them elsewhere?
    y
    • 2
    • 2
  • s

    Sandra Youssef

    05/04/2022, 12:27 AM
    It's finally out - Type-Safe Data Processing and Machine Learning Pipelines with Flyte and Pandera! From Data Council Austin by @Niels Bantilan

    https://www.youtube.com/watch?v=R-6Q2E9DT8Q&list=PLAesBe-zAQmEod2ARZjjAHmGFoGcjaXK6&index=33▾

    🚀 3
    :flyte: 2
    🔥 2
  • s

    Sandra Youssef

    05/04/2022, 2:54 PM
    Hi Flyers, Check out this quick intro to UnionML, an app that can be used to scale training jobs remotely and then connect to Flyte, presented by @Niels Bantilan. Also watch this week's community updates by @katrina.

    https://youtu.be/16uLQboGd3E▾

    https://youtu.be/BmDHqxhTTgA▾

    Mark your calendars for the next community sync on May 17th, which will feature guest speaker @jeev from Freenome. Details to come. See you all then! Flyte Team
    🔥 4
    🚀 6
  • m

    Matheus Moreno

    05/04/2022, 9:06 PM
    Hey, everyone. I updated flytekit and pyflyte to test the v1.0.0 and the new serialization process broke my code. The problem is that I have a decorator called
    @task_setup
    that, as the name suggests, executes some startup/teardown code before and after each task. This decorator is defined inside a submodule
    tasks.utils.flyte
    . My tasks are inside the
    tasks
    module, separated by different modules. When I packaged the tasks before, their names were `tasks.<module>.<task_name>`; now, their names are
    tasks.utils.flyte.<task_name>
    . Basically, the path for the decorator is being used as the "prefix" for the task. This is not ideal and the old method makes way more sense... What should I do?
    y
    k
    • 3
    • 46
  • m

    Matheus Moreno

    05/04/2022, 10:56 PM
    Also, is flytectl doing something special when starting the demo? Because I'm trying to start it using Docker and it takes too long... My Docker Compose file, which worked perfectly for the old
    flyte-sandbox
    (I just altered the ports):
    services:
      sandbox:
        image: <http://cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565|cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565>
        container_name: flyte-sandbox
        privileged: true
        ports:
          - "30080:30080"         # Console at <http://localhost:30080/console>
          - "30081:30081"
          - "30082:30082"
          - "30084:30084"
          - "30086:30086"
          - "30088:30088"
          - "30089:30089"
    y
    • 2
    • 15
  • a

    austin

    05/05/2022, 2:52 AM
    https://github.com/flyteorg/flyte/issues/2433 <- can someone help with this? Getting failure notices makes me not want to file issues 😕
    p
    y
    • 3
    • 2
  • w

    William Burke

    05/05/2022, 2:53 PM
    Hi! Do I need a specific version of helm/kubectl to install the helm charts from the flyte repo? I keep getting this error
    Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.securityContext): unknown field "fsGroupChangePolicy" in io.k8s.api.core.v1.PodSecurityContext
    d
    • 2
    • 5
  • s

    Sandra Youssef

    05/05/2022, 6:00 PM
    Hi Flyers, Learn how to create event-driven applications with AWS Lambda and Flyte in this blog, which demonstrates this using a fun and simple Neural Style Transfer model. Written by Union.ai's @Samhita Alla. https://lnkd.in/gFXatR38
    🔥 7
    👏 6
    👍 2
  • e

    Eduardo Apolinario (eapolinario)

    05/06/2022, 7:47 PM
    As promised, we're publishing a security advisory for a security vulnerability in flyteconsole. We recommend users to upgrade to flyteconsole v0.52.0 or greater ASAP (which corresponds to Flyte v0.19.4). As mentioned earlier, we yanked all affected versions from the official image repository. We've also reached out to affected folks individually and believe that all impacted deployments have been patched. For more details about the exploit, please visit https://github.com/flyteorg/flyteconsole/security/advisories/GHSA-www6-hf2v-v9m9.
    a
    • 2
    • 2
  • m

    Matheus Moreno

    05/06/2022, 8:01 PM
    Hey, everyone! I'm kinda confused on how to use custom images on a sandbox with Flyte 1.x.x. At first I thought the sandbox would pull the image from my local repository, but this doesn't seem to be the case, since I'm receiving the error
    containers with unready status: [ash8dlh4m294mnd9psbs-f62iya2y-0]|Back-off pulling image "flyte-example:1.0"
    when trying to execute a workflow, even though I have the flyte-example:1.0 image on my local repo. But, at the same time, it doesn't seem like I should have to build the image inside the sandbox container, right? The container from
    flytectl demo start
    doesn't even mount the working directory into
    /root
    . I can't find anything in the documentation that says how to do this...
    n
    m
    +2
    • 5
    • 17
  • l

    Leonardo Almeida

    05/06/2022, 8:10 PM
    Hello guys! I need some help. We are doing some testing with flyte in the AWS environment with ALB. But I'm having trouble trying to run the example code:
    import typing
    import pandas as pd
    import numpy as np
    
    from flytekit import task, workflow
    
    @task
    def generate_normal_df(n:int, mean: float, sigma: float) -> pd.DataFrame:
        return pd.DataFrame({"numbers": np.random.normal(mean, sigma,size=n)})
    
    @task
    def compute_stats(df: pd.DataFrame) -> typing.Tuple[float, float]:
        return float(df["numbers"].mean()), float(df["numbers"].std())
    
    @workflow
    def wf(n: int = 200, mean: float = 0.0, sigma: float = 1.0) -> typing.Tuple[float, float]:
        return compute_stats(df=generate_normal_df(n=n, mean=mean, sigma=sigma))
    When I run the code using the local cluster using flytectl demo start (k3s) it works. But in production environment using EKS and ALB does not work.
    pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2                                                                                     SIGINT(2) ↵  10055  17:06:22 
    {"asctime": "2022-05-06 17:06:31,287", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867591.287595583\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867591.287594125\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
    {"asctime": "2022-05-06 17:06:32,274", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867592.274550785\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867592.274550285\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
    Traceback (most recent call last):
      File "/usr/local/bin/pyflyte", line 8, in <module>
        sys.exit(main())
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/clis/sdk_in_container/run.py", line 471, in _run
        wf = remote.register_script(
      File "/usr/local/lib/python3.8/dist-packages/flytekit/remote/remote.py", line 536, in register_script
        upload_location, md5_bytes = fast_register_single_script(
      File "/usr/local/lib/python3.8/dist-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
        upload_location = create_upload_location_fn(content_md5=md5)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
        return super(SynchronousFlyteClient, self).create_upload_location(
      File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 40, in handler
        return fn(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 834, in create_upload_location
        return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
      File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNAVAILABLE
    	details = "failed to connect to all addresses"
    	debug_error_string = "{"created":"@1651867592.675517306","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1651867592.675516568","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
    Do you have any ideas? I'm using helm with values-eks.yaml
    y
    • 2
    • 22
  • h

    Hampus Rosvall

    05/09/2022, 11:59 AM
    I was looking at best practices with CI/CD & Flyte in your documentation a few weeks back but I can’t seem to find it anymore. Is there such a document available?
    s
    • 2
    • 3
  • s

    Sören Brunk

    05/09/2022, 12:35 PM
    Is there a way to configure the storage location/prefix for code when using the new fast registration without bucket credentials? It used to be stored under
    bucket-name/fast/
    before (not sure if I had configured it before, I can't find any config related to that). Now it's just stored in random paths under the bucket root.
    k
    • 2
    • 9
Powered by Linen
Title
s

Sören Brunk

05/09/2022, 12:35 PM
Is there a way to configure the storage location/prefix for code when using the new fast registration without bucket credentials? It used to be stored under
bucket-name/fast/
before (not sure if I had configured it before, I can't find any config related to that). Now it's just stored in random paths under the bucket root.
k

Ketan (kumare3)

05/09/2022, 2:31 PM
You can configure the prefix - https://github.com/flyteorg/flyteadmin/blob/072ce740118be58b57e8e6e26ea552368f31ac9e/pkg/config/config.go#L39
🙏 1
But the storage location is where Flyte stores all it's metadata
s

Sören Brunk

05/09/2022, 3:00 PM
That's fine, I just want to have it under a prefix, not the bucket root
Hmm perhaps I misunderstood how this works. If I set
storagePrefix: fast
, flyte still creates random directories at the root of the bucket, and THEN prepends the prefix like:
<gs://bucket/jl/fast/project/domain/><random>/<random>.tar.gz
Using an absolute prefix
storagePrefix: "<gs://bucket/fast>"
doesn't work either.
k

Ketan (kumare3)

05/25/2022, 2:31 PM
Ya we can stop doing the random can you file an issue
It's using the same sharded
s

Sören Brunk

05/25/2022, 2:33 PM
Ok I'll create an issue
I realized my issue is just about the prefix order, not sure about the random so I left that out for now. Here’s the issue: https://github.com/flyteorg/flyte/issues/2545
View count: 9