https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • d

    David Przybilla

    04/16/2022, 8:17 AM
    👋 Hey everybody, amazing project. Ive been learning and reading flyte's code. Is there any roadmap docs? so far I have only found https://github.com/orgs/flyteorg/projects/3/views/1 are there any recordings of the community syncs ? 🙂
    👋 3
    a
    k
    • 3
    • 3
  • a

    Alex Bain

    04/17/2022, 11:39 PM
    I just updated all my Flyte components to the new release
    v0.19.4
    (i.e. Flyte Console
    v0.52.1
    , Flyte Admin
    v0.6.147
    , etc). When I browse to Flyte Console, instead of getting a list of my projects I get:
    Failed to fetch data
    invalid wire type 4 at offset 1
    Retry (button)
    This looks like a gRPC binary incompatibility between the latest Flyte Console and Flyte Admin. Flyte team, could you take a look?
    ✅ 1
    k
    p
    +2
    • 5
    • 64
  • l

    Louis DiNatale

    04/18/2022, 2:35 PM
    Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?
    m
    p
    k
    • 4
    • 17
  • s

    Sandra Youssef

    04/18/2022, 6:02 PM
    Hi Flyers, There's a lot happening this week, so mark your calendars! For starters, join us in welcoming our guest speakers from Spotify - @Nelson Arapé & @Babis Kiosidis - who will join our Community Sync tomorrow 4/19 @9am PT, to talk about Leveraging Flyte for Frictionless Upgrades. Zoom Link Meeting Notes See you there! Flyte Team
    :flyte: 5
    😛artyparrot: 8
    🚀 5
    💯 5
    ❤️ 8
    🔥 11
  • a

    Alex Pozimenko

    04/18/2022, 6:30 PM
    hi team, happy Monday. We are getting
    terminated in the background, manually
    intermittently. No errors are reported in the POD log. Any suggestions on how to troubleshoot?
    e
    d
    k
    • 4
    • 25
  • r

    Rahul Mehta

    04/18/2022, 7:08 PM
    Hey folks, currently investigating Flyte as an alternative to our current setup which uses Kubeflow/Argo -- we have several pipelines with a very high degree of fan-out (on the order of ~700-1000 individual tasks). Does anyone have data points/benchmarks on how Flyte scales wrt the size of the workflow? Many of the features in the SDK (especially how flytekit works w/ native python types) look quite promising and could eliminate a lot of workarounds we've implemented to use Kubeflow.
    d
    k
    • 3
    • 17
  • r

    Rahul Mehta

    04/19/2022, 2:52 PM
    Would people be open to a PR that adds support for specifying the shasum of a docker image in addition to the tag? Seems like a pretty minor change to
    flytekit
    that'd enable specifying images like `docker.io/my-image@sha256:...`/is helpful to ensure that a specific workflow is locked to an exact image. Happy to make the PR later this week.
    y
    h
    • 3
    • 5
  • s

    Sandra Youssef

    04/20/2022, 6:08 AM
    Learn why frictionless upgrades are important at Spotify, and why Flyte was chosen as their new orchestration system, in Leveraging Flyte for Frictionless Upgrades by @Nelson Arapé.

    https://youtu.be/0OATtyM7xY8▾

  • s

    Sandra Youssef

    04/20/2022, 6:08 AM

    https://youtu.be/0OATtyM7xY8▾

  • s

    Sandra Youssef

    04/20/2022, 6:15 AM
    Hi Flyers, Today's Flyte Community Sync featured: • Leveraging Flyte for Frictionless Upgrades by @Nelson Arapé. Learn why frictionless upgrades are important at Spotify, and why Flyte was chosen as their new orchestration system.

    Recording▾

    • Community Updates.

    Recording▾

    Meeting Notes See you all on May 3rd!
    🤩 3
    p
    k
    • 3
    • 3
  • s

    Sandra Youssef

    04/20/2022, 6:23 AM
    And remember to register for and attend our Meetup with the MLOps Community, happening on Wednesday 4/20 at 9am PT. MLOps EngineeringLabs: What We Learned Building End-to-end ML Applications on Flyte Featuring: • @Haytham Abuelfutuh: How Flyte Orchestrates Tasks and Workflows • @Niels Bantilan: What We Learned Building End-to-End ML Applications on Flyte, featuring lightning talks by the 3 winning teams. https://home.mlops.community/public/events/mlops-engineeringlabs-what-we-learned-building-end-to-end-ml-applications-on-flyte-2022-04-20
  • c

    Calvin Leather

    04/20/2022, 1:54 PM
    Hey all, my team is shifting some existing data processing code over to flyte and we are using map tasks to replace some existing parallel code, had a question about
    functools.partial
    and its (possible) flyte equivalent
    . More details in thread
    k
    y
    +2
    • 5
    • 14
  • r

    Raviraja Ganta

    04/21/2022, 8:00 PM
    I am trying the example present in documentation. Getting the following error
    (flyte)  raviraja@ip-192-168-124-8  ~/Desktop/flows/flyte_flows    pyflyte run example.py:wf --n 500 --mean 42 --sigma 2
    
    Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
    Try 'pyflyte --help' for help.
    
    Error: No such command 'run'.
    (flyte)  raviraja@ip-192-168-124-8  ~/Desktop/flows/flyte_flows    pyflyte --help
    Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
    
      Entrypoint for all the user commands.
    
    Options:
      -c, --config TEXT       Path to config file for use within container
      -k, --pkgs TEXT         Dot separated python packages to operate on.
                              Multiple may be specified  Please note that this
                              option will override the option specified in the
                              configuration file, or environment variable
      -i, --insecure BOOLEAN  Disable SSL when connecting to Flyte backend.
      --help                  Show this message and exit.
    
    Commands:
      init         Create flyte-ready projects.
      local-cache  Interact with the local cache.
      package      This command produces a Flyte backend registrable package...
      serialize    This command produces protobufs for tasks and templates.
    run
    is not available. Am I missing something? When running normally with
    python example.py
    it is working fine. I have installed
    flytekit==0.32.6
    this version.
    g
    y
    • 3
    • 16
  • a

    Alex Bain

    04/21/2022, 9:30 PM
    Flyte team - do you think there will be a Flyte
    v0.19.5
    release (before the 1.0.0 release)? Either way, what do you think is a timeline for the next release version? Rough estimates ok.
    k
    • 2
    • 2
  • r

    Raviraja Ganta

    04/22/2022, 10:41 AM
    I tried installing all my dependenices in a docker image and provided that in remote execution
    (flyte)  raviraja@ip-192-168-124-8  ~/Desktop/python-monorepo/taxonomy/taxonomy/flows/flyte_flows   flyte_demo  pyflyte run --remote --image flyte_image:latest cluster_creation_flow.py:reason_clusters_pipeline
    In the UI it is coming as
    p
    • 2
    • 11
  • d

    David Przybilla

    04/24/2022, 3:58 AM
    https://github.com/flyteorg/flyte/issues/317 I wonder if it would be ok to make public the design for flyte copilot doc mentioned in that issue ? . Context: I am just trying to understand more about flyte and contribute if possible .
    ❤️ 1
    s
    k
    a
    • 4
    • 19
  • s

    Samhita Alla

    04/25/2022, 12:25 PM
    📣 Contributors of the Month 📣 Hello everyone! We’re fortunate enough to have contributors who not only strive to make Flyte better but help us reach a wider audience. On that note, we want to give shout-outs and send swag to the following list of contributors this month. • @Konstantin Slisenko has written a comprehensive article on orchestrating data pipelines at Lyft; this is the go-to article for those who’d like to know how Flyte differs from and complements Airflow. • @Guillaume Perchais and @Babis Kiosidis have published an article detailing why Spotify switched the data orchestration service from Luigi and Flo to Flyte. They’ve also extensively contributed to flytekit-java, our Java SDK. We hope to have many more of your contributions, Konstantin, Guillaume, and Babis! All contributors of the month should be available on flyte.org.
    🎉 9
    🏅 5
    🙌 2
    🔥 11
  • m

    Mike Ossareh

    04/25/2022, 3:20 PM
    the assimilation is complete. I now type "flyte" when I mean to type "flight" 🤦‍♂️🏻
    😆 12
    🧠 1
    g
    • 2
    • 1
  • m

    Mike Zhong

    04/25/2022, 4:15 PM
    Quick question about using dynamic workflows with the imperative style
    create_node()
    . We have a bunch of
    @tasks
    which have dependencies on one another, but they are not explicit (nothing is returned or passed in). We can easily use the imperative style to define the order of the tasks in a regular
    @workflow
    . Does the imperative style also work for
    @dynamic
    workflows?
    s
    y
    • 3
    • 10
  • a

    Alex Bain

    04/26/2022, 1:59 AM
    Bad news @Nastya Rusina, I bounced to Flyte Console
    v0.55.0
    and I'm back to
    Failed to fetch data: invalid wire type 4 at offset 1
    for
    /console
    n
    k
    c
    • 4
    • 5
  • a

    Anil Kumar Nandibhatla

    04/26/2022, 3:58 AM
    👋 Hi everyone!
    👋 2
    k
    • 2
    • 1
  • h

    Hafsa Junaid

    04/26/2022, 6:09 AM
    Hey team! I am configuring K8s spark operator on GCP. Do we have a replacement of the below config for GCP??
    s
    y
    • 3
    • 2
  • d

    Dylan Wilder

    04/26/2022, 4:19 PM
    I know i'm late to the party, but congrats all on the funding round!! 🎉
    🎉 9
    🚀 6
  • s

    Sandra Youssef

    04/26/2022, 6:06 PM
    Hi Flyers, Union.ai is going to PyCon Startup Row & Job Fair this week, representing the Flyte community! Meet the founders of Union.ai and check out Flyte's Quickdraw Recognition demo! Expo Hall: April 28-30th / Job Fair: May 1st https://us.pycon.org/2022/
    🐍 4
    ✏️ 5
  • s

    Sandra Youssef

    04/26/2022, 7:12 PM
    🥁🥁🥁 And.... it's out! 🎉*Flyte v1.0*🎉 is our first major release! Flyte APIs, and tools can now be considered [stable]. Most, if not all, workflows/tasks written against FlyteKit
    v1.0.0
    should be expected to run against all
    v1.x.x
    versions. Features include, among others: • Single Binary (Fast Flyte deployment) and improved Garbage Collector [Core] • File Ignore and Script Mode (Single command to register and run workflows) [FlyteKit] • Project dashboard, Map task reporting, and Dynamic workflow rendering [FlyteConsole] Check out the full list here: https://github.com/flyteorg/flyte/releases/tag/v1.0.0 🎉🎉🎉
    ❤️ 11
    🍾 7
    🚀 12
    :flyte: 18
    ⚡ 7
    🎉 26
    😛artyparrot: 15
    🔥 6
    1️⃣ 9
  • b

    Bernhard Stadlbauer

    04/27/2022, 11:43 AM
    Hi! We are currently trying to fast-register workflows using
    flytectl==0.5.21
    . The workflows were serialized using
    pyflyte
    (from `flytekit=1.0.0`; using
    fast
    ). The storage configuration looks like this:
    ... 
    storage:
      type: stow
      stow:
        kind: google
        config:
          project_id: "<project>"
          scopes: <https://www.googleapis.com/auth/devstorage.read_write>
      container: "<bucket>"
    Now, when running
    flytectl register files tmp/workflows/* -p acorn -d development
    , we see:
    Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
    {"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}
    GOOGLE_APPLICATION_CREDENTIALS
    env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0
    p
    h
    • 3
    • 19
  • a

    austin

    04/27/2022, 3:05 PM
    Is there a github issue tag for issues related to user onboarding? If a current priority of flyte is to increase ecosystem size, then helping ease user onboarding would likely be important. It does not seem there are relevant tags to highlight issues related to such ends?
    y
    • 2
    • 7
  • a

    austin

    04/27/2022, 3:26 PM
    what’s the difference between boilerplate --> https://github.com/flyteorg/boilerplate and https://github.com/flyteorg/datacatalog/tree/master/boilerplate … would have expected https://github.com/flyteorg/boilerplate to be the place where boilerplate gets updated, and the other places to be propagated from there. the datacatalog location appears to have more recent updates, so perhaps my assumption/understanding is incorrect?
    n
    k
    • 3
    • 9
  • a

    Alex Bain

    04/27/2022, 5:28 PM
    We just updated from Flyte release
    v0.19.3
    to new
    v1.0.0
    and Flyte Admin no longer wants to register workflows like:
    @flytekit.workflow
    def DDPTrainingWorkflow(
        train_datasets: typing.List[str] = [],
        ...
    ):
        patch_experiment(train_datasets)
    
    @flytekit.task()
    def patch_experiment(
        train_datasets: typing.List[str],
    )
    ^^^^ while registering the workflow, this throws:
    [2022-04-27T15:09:25Z] 	details = "Type mismatch for Parameter train_datasets in default_inputs has type collection_type:<simple:STRING > , expected simple:NONE "
    clearly something doesn't like the default
    []
    value of
    train_datasets
    against its declared type
    typing.List[str]
    . Ideas?
    k
    h
    +2
    • 5
    • 31
  • k

    katrina

    04/27/2022, 6:41 PM
    FYI, due to a security vulnerability all users must upgrade their flyteconsole images to ghcr.io/flyteorg/flyteconsole:v0.52.0 or greater. all previous image versions of the console have been yanked. we've done our best to reach out with folks individually to ensure they have upgraded their deployments but please upgrade if you have not already done so and by next friday! once the security embargo is over, we will be publishing a CVE on May 6 with an explanation of what happened. if you'd like to be added to the CVE, please DM me your github handle. thanks folks for your continued support 🙏
    👍 2
    💯 2
Powered by Linen
Title
k

katrina

04/27/2022, 6:41 PM
FYI, due to a security vulnerability all users must upgrade their flyteconsole images to ghcr.io/flyteorg/flyteconsole:v0.52.0 or greater. all previous image versions of the console have been yanked. we've done our best to reach out with folks individually to ensure they have upgraded their deployments but please upgrade if you have not already done so and by next friday! once the security embargo is over, we will be publishing a CVE on May 6 with an explanation of what happened. if you'd like to be added to the CVE, please DM me your github handle. thanks folks for your continued support 🙏
👍 2
💯 2
View count: 1