https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • s

    Sandra Youssef

    07/22/2022, 3:22 PM
    Hi everyone, The countdown has begun! ⚡Flyte will be featured in Kelsey Hightower's Twitter Space: Machine Learning in Production in about 90 minutes⚡ Join @Ketan (kumare3) and @Eduardo Apolinario (eapolinario), plus guest speakers @Fabio Grätz, @Evan Sadler, @Arno Hollosi, @Jordan Ganoff and @Gautam Kedia for insights into unified machine learning and data orchestration. 7/22 at 10am PT Event link (also QR code in the flyer)
    👏 21
    j
    • 2
    • 2
  • s

    Sandra Youssef

    07/22/2022, 3:23 PM
    Feel free to share and repost!
  • s

    Sandra Youssef

    07/22/2022, 6:26 PM
    The recording is available using the same event link: https://twitter.com/i/spaces/1ZkKzbXLekWKv
  • s

    Samhita Alla

    07/25/2022, 11:22 AM
    📣 Contributors of the Month 📣 Hi everyone! For the month of July, we want to give shout-outs and send swag to the following contributors who are sweating it out to make Flyte even better! • @Sugato Ray made
    conda-forge
    packages for unionml, flytekit, and flyteidl. • @Robin Kahlow worked on a cookie cutter template that simplifies the creation and execution of flyte workflows. Do check out the repository! • @Babis Kiosidis worked on performance improvements as he was seeing elevated
    RESOURCE_EXCHAUSTED
    when writing various files from flytepropeller to GCS. It’ll hopefully be merged very soon into flytepropeller! We hope to have many more of your contributions, Sugato, Robin, and Babis! All contributors of the month should be available on flyte.org.
    👏 13
    👍 1
  • s

    Sandra Youssef

    07/25/2022, 9:52 PM
    Hi Flyers, This week's Flyte Community Sync will feature a Biotech special! Meet industry practitioners and tech leads: • @Kenny Workman from LatchBio • @jeev from Freenome • @Calvin Leather from Embark Veterinary • @krishna Yerramsetty from Infinome Biosciences, and • @Jason Daniel from Zymergen In addition to our own @Ketan (kumare3) and @Niels Bantilan. Hear their stories, followed by a Q&A. 7/26 at 9am PT. https://addevent.com/event/rT7842495
    💯 4
    ❤️ 8
    🔥 4
  • a

    Ailin Yu

    07/26/2022, 12:39 AM
    Hello, I have a super basic question! I’d like to run a local flyte cluster to test with, which I thought I would do by running
    flyte start --config=flyte_local_k3d.yaml
    , (after having run
    make compile
    already) for example when using k3d. Even though my
    .k3d/kubeconfig-flyte.yaml
    config file works with k3d directly, when I try to run the above command I always get an error:
    {"json":{"src":"start.go:113"},"level":"error","msg":"Failed to create controller manager. error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"} {"json":{"src":"start.go:164"},"level":"panic","msg":"Failed to start Propeller, err: error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"}
    Am I using this incorrectly or do I need to have something else in my k3d yaml config perhaps?
    d
    • 2
    • 4
  • s

    Sören Brunk

    07/26/2022, 6:22 AM
    Hi, I was surprised that my long running interruptible training task was scheduled onto a non-spot instance after a failure (probably due to preemption). Thinking about it, it makes sense as default behavior, and is also documented:
    If your task gets preempted, Flyte will retry your task on a non-spot (regular) instance. This retry will not count towards a retry that a user sets.
    Still I'd like to configure Flyte to try one more time on a spot instance. There's a interruptible-failure-threshold ("number of failures for a node to be still considered interruptible"). Is this the right config to tweak that behavior?
    j
    d
    • 3
    • 3
  • j

    jeev

    07/26/2022, 5:10 PM
    thanks for that great panel discussion on flyte in biotech! @Martin Stein @Ketan (kumare3) @Niels Bantilan @Sandra Youssef and for the great insights from my fellow biotech engineers/scientists: @Calvin Leather @Jason Daniel @Kenny Workman @krishna Yerramsetty for anyone that is considering flyte for biotech (or anything for that matter), please feel free to reach out! looking forward to growing this community!
    ❤️ 10
    💯 1
    c
    k
    k
    • 4
    • 10
  • s

    Sandra Youssef

    07/26/2022, 6:01 PM
    Hi Flyers, The recording of @Ketan (kumare3)’s podcast with the MLOPs community is out!

    Why You Need More Than Airflow▾

    Learn why we need a fabric for ML orchestration, and how to build that layer correctly. https://mlops.community/watch/why-you-need-more-than-airflow_qw3XkiPuPJFeQ9/
    👍 10
  • b

    Bernhard Stadlbauer

    07/27/2022, 10:26 AM
    Multi-cluster flyte permissions question 🧵
    k
    • 2
    • 8
  • m

    Matheus Moreno

    07/27/2022, 5:45 PM
    Hi, everyone. Quick question: is the
    --force
    flag on
    flytectl register files
    working? When I use it, the CLI logs "AlreadyExists", which makes it look like it didn't actually rewrite anything. Is this normal? If so, shouldn't the logs print another message, such as "Overwritten" or something? I'm using FlyteCTL 0.5.6, by the way.
    y
    p
    • 3
    • 16
  • s

    SeungTaeKim

    07/28/2022, 1:36 AM
    Hi, All! I would like to check which users run executions in Flyte remote server, but I guess Flyte does not provide this, am I right? I cannot see any logs who executes workflow from kubernetes. (my MLOps team is about to release Flyte to all research engineer of my company, so I want to check this point)
    k
    • 2
    • 5
  • t

    Tom Szumowski

    07/29/2022, 12:39 PM
    Newcomer question. I was wondering if anyone had tips on how to configure flyte to access GPUs. I see this document: https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/configure_use_gpus.html#sphx-glr-auto-deployment-configure-use-gpus-py But I am curious how to apply that configuration shown:
    plugins:
      k8s:
        resource-tolerations:
          - <http://nvidia.com/gpu:|nvidia.com/gpu:>
            - key: "key1"
              operator: "Equal"
              value: "value1"
              effect: "NoSchedule"
    I saw this thread which mentioned updating the propellor configmap. Is that the way to go? Or is there a way to update plugins via the flyte config? Thanks!
    y
    s
    • 3
    • 9
  • m

    Matheus Moreno

    07/29/2022, 6:23 PM
    Hi, everyone. I'm trying to test out the BigQuery plugin using the example in the documentation, but when I try to convert the StructuredDataset into a DataFrame, I get this error:
    Traceback (most recent call last):
          ...
          File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/types/structured/basic_dfs.py", line 63, in decode
            ctx.file_access.get_data(path, local_dir, is_multipart=True)
          File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
            raise FlyteAssertion(
    
    Message:
    
        Failed to get data from  to /tmp/flyteltak17mx/local_flytekit/85e6844a8f8283f8841994437be73626 (recursive=True).
    
    Original exception: cannot copy tree '': not a directory
    
    User error.
    No path for the resulting query file. Could it be that there was an error executing the task? Flyte says the task succeeded, but I can't check on the GCP console (I don't have the necessary permissions).
    k
    y
    • 3
    • 62
  • t

    Tao He

    08/02/2022, 3:51 AM
    Hi folks, I have read [the excellent tutorial about distributed training](https://docs.flyte.org/projects/cookbook/en/stable/auto/case_studies/ml_training/spark_horovod/keras_spark_rossmann_estimator.html#). I'm wondering whether the sentence "'SparkBackend' uses
    horovod.spark.run
    to execute the distributed training function" means that flyte will launch
    num_proc
    spark workers on
    num_proc
    flyte worker? Or it just launches them on one spark worker?
    k
    • 2
    • 14
  • t

    Tao He

    08/02/2022, 3:52 AM
    If the former is the case, then how does flyte decide to launch spark workers on which set of flyte workers?
  • n

    Niels Bantilan

    08/02/2022, 12:42 PM
    anyone here want to put their 2 cents into this HN Airflow discussion? https://news.ycombinator.com/item?id=32317558 😀
    k
    • 2
    • 1
  • k

    Ketan (kumare3)

    08/02/2022, 10:02 PM
    service roles can be configured at 3 levels 1. Project + domain defaults (every execution launched in this project / domain will use this SA) 2. launchplan default. (every invocation of this launchplan will use this SA) 3. Execution time override (override at invocation only for a specific execution)
    s
    • 2
    • 2
  • r

    Robert Everson

    08/02/2022, 10:55 PM
    somewhat related to this https://flyte-org.slack.com/archives/CNMKCU6FR/p1657727959446239 Are all notifications that come out of flyte when a workflow fails the same (from this config)? For example, I have a project that would like to customize the content of the failure notifications that get sent on a per task/workflow basis.
  • t

    Tao He

    08/04/2022, 7:32 AM
    Hi, folks, I have encountered many pieces about project "flytectldemo" in the flytectl's documentation (e.g., https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_create_execution.html). However, I failed to find the repo and
    flytectl register examples
    doesn't register such a project as well. May I know where can I find that? Or it is just because the flytectl's doc is out-of-date?
    k
    • 2
    • 5
  • k

    Ketan (kumare3)

    08/04/2022, 3:30 PM
    Folks we need help, we are working on some core changes that will potentially improve performance of the core engine upto 100x, so a pure cached workflow will complete in less than a second, all transitions will be faster. Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows. We would love volunteers who can help us test and benchmark it? Highly appreciated
    🔥 5
    💯 4
    r
    s
    +4
    • 7
    • 28
  • n

    Nada Saiyed

    08/05/2022, 12:41 AM
    Hello! What’s the best way to define a task config that launches a custom SageMaker hyperparameter tuning job. I am trying to create a task similar to this example. i see i can provide the
    task_config
    as a
    HPOJob
    but how do i specify which
    training_task
    it’s supposed to execute? and also where should i specify the parameters like
    HyperparameterTuningJobConfig
    , and
    ParameterRanges
    ?
    s
    k
    +2
    • 5
    • 19
  • s

    Sujith Samuel

    08/05/2022, 5:25 AM
    Hello #general
  • s

    Sujith Samuel

    08/05/2022, 5:25 AM
    I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public) and flytepropeller(access type confidential with client Id and secret). I am following the keycloak section in the https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html# My relevant values in the helm values file are as below secrets: adminOauthClientCredentials: # -- If enabled is true, helm will create and manage
    flyte-secret-auth
    and populate it with
    clientSecret
    . # If enabled is false, it's up to the user to create
    flyte-secret-auth
    as described in # https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server enabled: true clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C" clientId: "flytepropeller" -- FlyteAdmin server configuration adminServer: # Refer to the server config. server: httpPort: 8088 grpcPort: 8089 security: # -- Controls whether to serve requests over SSL/TLS. secure: false # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. useAuth: true allowCors: true allowedOrigins: # Accepting all domains for Sandbox installation - "*" allowedHeaders: - "Content-Type" - "flyte-authorization" # Refer to the full structure for documentation. flyteadmin: roleNameKey: "iam.amazonaws.com/role" profilerPort: 10254 metricsScope: "flyte:" metadataStoragePrefix: - "metadata" - "admin" eventVersion: 2 testing: host: http://flyteadmin # -- Authentication configuration auth: authorizedUris: # This should point at your public http Uri. - https://flytedeployment url # This will be used by internal services in the same namespace as flyteadmin - http://flyteadmin:80 # This will be used by internal services in the same cluster but different namespaces - http://flyteadmin.nmlp.svc.cluster.local:80
    # Controls app authentication config
      appAuth:
        thirdPartyConfig:
          flyteClient:
            clientId: flytectl
            redirectUri: https://<flyte deployment url>/callback
            scopes:
              - offline
              - all
    
      # Controls user authentication
      userAuth:
        openId:
          baseUrl: https://<keycloak production realm>/realms/nsdmlp
          scopes:
            - profile
            - openid
          clientId: flytepropeller
    I have also edited the flyte-admin-secrets to have the correct client secret apiVersion: v1 data: claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw== cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw== cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE= oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T18:34:11Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T18:34:12Z"} Logs from 8/5/2022, 12:02:39 AM And the chrome errors out with 403 access denied page. I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error. Please assist. Thanks, Sujith
    k
    s
    +2
    • 5
    • 86
  • e

    Edgar Trujillo

    08/05/2022, 2:44 PM
    Hey All... qq around the runtime env of a flyte workflow. In Quickly Re-deploy Your Application, there's a note:
    --fast flag will take the code from your local machine and provide it for execution without having to build the container and push it. The --force flag allows overriding your previously created package.
    How does flyte decide what to push up and how does the updated code get packaged into the container without a rebuild?
    s
    • 2
    • 1
  • a

    Alex Pozimenko

    08/05/2022, 5:30 PM
    hi team, the documentation states that subworkflows execute within context of the parent workflows (here). What are the implications of that?
    k
    • 2
    • 9
  • a

    Alex Pozimenko

    08/05/2022, 11:17 PM
    flytekit q:
    execution = flyte_remote.fetch_execution(name = execution_id)
    print (execution.node_executions)
    is
    execution.node_executions
    expected to be populated? If not, how do I get the node execution details? I'm looking for level of details similar to returned by
    flytectl get execution execid --details --nodeID n0 -o yaml
    t
    • 2
    • 7
  • o

    Oliver Nguyen

    08/08/2022, 5:10 AM
    Hi everyone. I'm trying to use a scheduled workflow in Flyte and I have some questions. 1. is there any way to deactivate the launch plan? 2. a new pod will be created every time the scheduler runs, is there a limit to the amount of pods created by the scheduler? Thanks
    p
    k
    • 3
    • 6
  • a

    Abdullah Mobeen

    08/08/2022, 1:53 PM
    .
  • r

    Rahul Mehta

    08/08/2022, 5:56 PM
    Hi, is there a way to specify a human-readable name for an execution? I notice
    --name
    is an available flag in
    pyflyte run
    , but couldn't find the equivalent for
    flytectl create execution
    . Mostly looking for how a user could surface a more informative name in the UI vs the randomly-generated execution ID
    k
    • 2
    • 7
Powered by Linen
Title
r

Rahul Mehta

08/08/2022, 5:56 PM
Hi, is there a way to specify a human-readable name for an execution? I notice
--name
is an available flag in
pyflyte run
, but couldn't find the equivalent for
flytectl create execution
. Mostly looking for how a user could surface a more informative name in the UI vs the randomly-generated execution ID
k

Ketan (kumare3)

08/08/2022, 9:42 PM
i think that is the name
r

Rahul Mehta

08/08/2022, 9:43 PM
I specified
--name
when creating an execution using
pyflyte run
(ie.
--name foo
) and didn't see this surfaced anywhere in the UI. Am I misunderstanding the purpose of that flag?
k

Ketan (kumare3)

08/08/2022, 10:03 PM
hmm this should be the execution name
this is used to de-dupe
@Rahul Mehta you are right - this should be passed in as a parameter - https://github.com/flyteorg/flytectl/blob/2c274d126889dc1f6f5de2677eddadf486910347/cmd/create/execution_util.go#L149
would you like to propose a change?
currently it is only generated, but that should be a fallback
View count: 9