https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • j

    jeev

    02/28/2022, 8:31 PM
    seeing a weird issue with flyteadmin when upgrading from 0.6.44 -> 0.6.104:
    time="2022-02-28T20:30:40Z" level=info msg="Using config file: [/etc/flyte/config/allowed_origins.yaml /etc/flyte/config/cluster_resources.yaml /etc/flyte/config/core.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage-limits.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
    {"json":{"src":"migrate.go:107"},"level":"fatal","msg":"failed to connect to `host=cloud-sql-proxy user=flyte database=flyte`: server error (ERROR: SSL required (SQLSTATE 08P01))","ts":"2022-02-28T20:30:40Z"}
    
    2022/02/28 20:30:40 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/database.go:102
    [error] failed to initialize database, got error failed to connect to `host=cloud-sql-proxy user=flyte database=flyte`: server error (ERROR: SSL required (SQLSTATE 08P01))
    y
    h
    +2
    • 5
    • 24
  • j

    jeev

    03/02/2022, 8:56 PM
    are there ongoing discussions about vertical pod scaling in flyte to handle OOMs?
    k
    • 2
    • 3
  • k

    Ketan (kumare3)

    03/04/2022, 4:38 PM
    Please submit talks for https://events.linuxfoundation.org/open-source-summit-north-america/program/cfp/
    f
    s
    • 3
    • 14
  • s

    Sören Brunk

    03/07/2022, 7:56 PM
    I'm having issues when running a fast-registered workflow. It looks like flytekit (0.30.3) can't find the code archive:
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    
    [vej152kn4b-n1-0] terminated with exit code (1). Reason [Error]. Message: 
    tar: /root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz: Cannot open: No such file or directory
    tar: Error is not recoverable: exiting now
    Traceback (most recent call last):
      File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
        sys.exit(fast_execute_task_cmd())
      File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 467, in fast_execute_task_cmd
        _download_distribution(additional_distribution, dest_dir)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 112, in download_distribution
        result.check_returncode()
      File "/usr/lib/python3.9/subprocess.py", line 460, in check_returncode
        raise CalledProcessError(self.returncode, self.args, self.stdout,
    subprocess.CalledProcessError: Command '['tar', '-xvf', '/root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz', '-C', '/root']' returned non-zero exit status 2.
    .
    Here's what's executing from the pod spec
    spec:
      containers:
      - args:
        - pyflyte-fast-execute
        - --additional-distribution
        - <gs://my-bucket/fast/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz>
    ...
    I checked the bucket and the object exists so uploading during registration seems to be working correctly. Perhaps something wrong with the download or the path it is copied to in the container. Any ideas?
    k
    y
    • 3
    • 23
  • s

    Sören Brunk

    03/09/2022, 10:15 PM
    Hey flyte friends, sorry for the spam but I'm running into yet another weird error. Might be some misconfiguration on my side as it used to work but I wasn't able to figure out yet what change causes the issue. So what's happening is that suddenly mosts tasks fail almost immediately after they started running. The high-level error shown in flyteconsole is this:
    Some node execution failed, auto-abort.
    The task pods start, go into running state. I can see a few lines of task logs sometimes but then it looks like the pod is just deleted long before it's finished. It terminates and is cleaned up. Log says:
    Stopping container m4dydd0r8y-n0-0
    So i checked flytepropeller logs and there's one error log a few seconds before the pod is deleted. Not sure if it's related:
    Failed to update workflow. Error [Operation cannot be fulfilled on <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> "m4dydd0r8y": the object has been modified; please apply your changes to the latest version and try again]
    I'm a bit at a loss here how to debug this further. Any hints much appreciated. I can also provide the full propeller log if that could help tracing this down.
    h
    • 2
    • 8
  • s

    Sandra Youssef

    03/14/2022, 5:30 PM
    Many thanks to @Guillaume Perchais and the Spotify team for publishing this wonderful blog about why they made the switch to Flyte: Why We Switched Our Data Orchestration Service Much appreciated and always a pleasure to work with you! Flyte Team
    🔥 8
    :flyte-2: 4
    ❤️ 14
    g
    s
    +4
    • 7
    • 20
  • s

    Sandra Youssef

    03/15/2022, 5:04 PM
    Hi Flyers, Flyte Monthly - Issue #6 is out! Read about partner success stories and positive testimonials! Get a sneak-peek into upcoming community sync events, guest speakers, and meetups. Learn about our latest release and how we manage our workflows. Subscribe here for the full scoop: https://www.getrevue.co/profile/flyte
    m
    • 2
    • 6
  • j

    Joseph Wang

    03/16/2022, 7:36 PM
    is it possible to define subworkflow, which gets included as part of wf def
    y
    k
    • 3
    • 15
  • k

    Ketan (kumare3)

    03/17/2022, 12:07 AM
    I realized that a lot of folks do not have an authoratative reference for Flyte related stuff, that they can present at their organizations. Here is a working copy of a presentation that I use sometimes - Flyte
    👍 1
    p
    • 2
    • 1
  • k

    Ketan (kumare3)

    03/17/2022, 4:02 AM
    As usual its that time of week. I am on the zoom link, here to help answer any questions, discuss ideas etc - Office hours - cc @Serkan Unal
  • k

    komal azram

    03/17/2022, 6:19 AM
    Hi, can anyone tell me what's the datatype for spark data frame in flyte?
    y
    • 2
    • 14
  • s

    Sören Brunk

    03/17/2022, 10:10 AM
    Quick question about flyte remote. I have a workflow that produces a
    FlyteDirectory
    as final output. Via flyteconsole/API I get a reference to the object, like so:
    o0:
      type:
        multi-part
      uri:
        <gs://flyte-bucket/data/q1/pjziwgu55i-n1-0/48f7d1af2f65a6ff4c03fac9191cbea9>
    If I try to fetch that workflow via
    FlyteRemote
    I get a local path as output. It's empty though.
    remote = FlyteRemote.from_config()
    exec = remote.fetch_workflow_execution("abc")
    print(exec.outputs)
    // /tmp/flyte3eg_mcpz/control_plane_metadata/local_flytekit/6f2223e1c864f6d1e88c8dbad090f349
    What is the purpose of this local path and is there a way to get the original output location via
    FlyteRemote
    ?
    k
    y
    t
    • 4
    • 12
  • h

    Hafsa Junaid

    03/17/2022, 7:39 PM
    Hey! Why is it necessary to install flytekit in virtual environment. ??? Flytekit not found error occurs even if the module is installed. But it works fine if flytekit is installed in venv and that activated venv is used for workflows and programs.
    y
    a
    • 3
    • 40
  • j

    Joseph Wang

    03/18/2022, 3:36 PM
    are there examples of branching and subwf in the repo
    e
    • 2
    • 2
  • j

    Joseph Wang

    03/18/2022, 3:38 PM
    do we support hp, exploration, submodel, etc pattern
    k
    • 2
    • 1
  • j

    Joseph Wang

    03/18/2022, 3:38 PM
    while continue(...):
       next_candidate_sets = get_next_candidate_set(...)
       results = run_jobs(next_candidate_sets)
       store_results(results)
  • j

    Joseph Wang

    03/18/2022, 5:19 PM
    any doc on how java & scala binding is done
    k
    • 2
    • 6
  • j

    Joseph Wang

    03/18/2022, 5:21 PM
    how about data serde btw different frameworks and langs
  • s

    Sandra Youssef

    03/18/2022, 6:16 PM
    DataCouncil Austin here we come! Learn about Type-Safe Data Processing and Machine Learning Pipelines with Flyte and Pandera, from Union.ai's @Niels Bantilan. March 23rd, 4pm CT.
    🎉 5
    🔥 2
  • m

    Mike Seid

    03/18/2022, 6:23 PM
    Has anyone thought about TFX and Flyte?
    k
    k
    • 3
    • 14
  • k

    Ketan (kumare3)

    03/18/2022, 9:30 PM
    @Sören Brunk have you started working on KerasCheckpoint wrapper? cc @Niels Bantilan?
    s
    • 2
    • 3
  • k

    Ketan (kumare3)

    03/20/2022, 4:33 AM
    I got some DM questions from @raajay viswanathan, wanted to share it more generally to share the answers these are the questions i have 1. Similar to Spark plugin do you have a Ray plugin? 2. Is there a way for launch plan of one workflow to use the output of a prior launch of another workflow? 3. Do you have a catalog for the data that flows between tasks? For example, if one task outputs a model (TF, Spark Model) can another service access it to deploy? Answers: 1. Not yet. We planned to, but not enough demand yet. I think we should probably sometime this year 2. Not directly. you mean explicit Workflow chaining? We call it reactive pipelines. Again not yet 3. Yes you can. this is what we have in Flyteremote. and this is the intended usecase.
  • h

    Hampus Rosvall

    03/20/2022, 8:35 PM
    Hey, I started using Flyte today and I really like it! I am experimenting with some workflows that outputs models to a MinIO S3 bucket using the Sandbox deployment. It is not quite clear to me how I query the artifacts in the MinIO S3 bucket e.g., given this output from the workflow - how do I access the MinIO bucket to acquire the model artifact?
    model_dir:
      type:
        multi-part
      uri:
        <s3://my-s3-bucket/ja/hfnqehz4ps-n0-0/9828036b2328e522488b3ff0311528e2>
    k
    y
    • 3
    • 18
  • h

    Hampus Rosvall

    03/20/2022, 8:35 PM
    message has been deleted
  • s

    Samhita Alla

    03/21/2022, 9:44 AM
    We reached a new milestone: Over 2,000 stars on GitHub. ⭐*️* Flyte wouldn’t have thrived without your constant support and feedback. We are indebted to each of you and so very grateful for all the love! 🙏
    😛artyparrot: 6
    🎉 12
    🤩 1
  • s

    Sandra Youssef

    03/21/2022, 5:46 PM
    Hi Flyers, Want to learn more about Flyte? Union.ai's @Ketan (kumare3) will talk about "*Intra-task Checkpointing & Flyte Decks*" at tomorrow's Flyte Open Source Community Sync. Tuesday March 22, 9am PT. Zoom link & calendar invite: https://www.addevent.com/event/EA7823958 See you there! Flyte Team
    🙌 6
    h
    k
    • 3
    • 5
  • k

    komal azram

    03/22/2022, 3:47 PM
    Hi I am trying to return a spark data frame but having this issue _user_exceptions.FlyteAssertion( flytekit.exceptions.user.FlyteAssertion: When calling tasks, only keyword args are supported. Aborting execution as detected 1 positional args (FlyteSchema.__class_getitem__.<locals>._TypedSchema(remote_path='/tmp/flyte/20220322_204000/raw/82eaa7c6b77680e044d794beaf4e01cb'),) I have defined column names with their data types as my_schema = FlyteSchema[kwtypes(ID=str,START_T=datetime.datetime,STOP=datetime.datetime,PATIENT=str,ENCOUNTER=str,CODE=str,intDESCRIPTION=str,REASONCODE=int,REASONDESCRIPTION=str)]. Following this https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/dataframe_passing.html guide
    k
    y
    s
    • 4
    • 55
  • k

    Ketan (kumare3)

    03/22/2022, 4:01 PM
    We are beginning our Bi Weekly OSS sync in 2-5 minutes. Please join. Zoom link https://zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09
  • h

    Hafsa Junaid

    03/22/2022, 5:00 PM
    Hi! I was to access flyteconsole service externalize endpoint. I have followed the complete document: https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual The workload and service are up (the screenshot is shared): • The docment says --> Use the https://<FLYTE-ENDPOINT>/console to get access to flyteconsole UI I have done the same and used the domain as flyte-endpoint but nothing is showing up. Are there any further steps to access flyteconsole UI ?? like create external IP instead of cluster IP or mapping service to domain
    k
    • 2
    • 1
  • f

    Fabian Baier

    03/22/2022, 8:36 PM
    Hi team, where can I ask an observability question about the Flyte user dashboard in grafana?
    k
    • 2
    • 1
Powered by Linen
Title
f

Fabian Baier

03/22/2022, 8:36 PM
Hi team, where can I ask an observability question about the Flyte user dashboard in grafana?
k

Ketan (kumare3)

03/22/2022, 8:43 PM
anywhere is fine, but deployment sounds right?
👍 1
View count: 6