Hi everyone, The countdown has begun! ⚡Flyte will be featured in Kelsey Hightower's Twitter Space: Machine Learning in Production in about 90 minutes⚡ Join @Ketan (kumare3) and @Eduardo Apolinario (eapolinario), plus guest speakers @Fabio Grätz, @Evan Sadler, @Arno Hollosi, @Jordan Ganoff and @Gautam Kedia for insights into unified machine learning and data orchestration. 7/22 at 10am PT Event link (also QR code in the flyer)

Feel free to share and repost!

The recording is available using the same event link: https://twitter.com/i/spaces/1ZkKzbXLekWKv

📣 Contributors of the Month 📣 Hi everyone! For the month of July, we want to give shout-outs and send swag to the following contributors who are sweating it out to make Flyte even better! • @Sugato Ray made

conda-forge

packages for unionml, flytekit, and flyteidl. • @Robin Kahlow worked on a cookie cutter template that simplifies the creation and execution of flyte workflows. Do check out the repository! • @Babis Kiosidis worked on performance improvements as he was seeing elevated

RESOURCE_EXCHAUSTED

when writing various files from flytepropeller to GCS. It’ll hopefully be merged very soon into flytepropeller! We hope to have many more of your contributions, Sugato, Robin, and Babis! All contributors of the month should be available on flyte.org.

Hi Flyers, This week's Flyte Community Sync will feature a Biotech special! Meet industry practitioners and tech leads: • @Kenny Workman from LatchBio • @jeev from Freenome • @Calvin Leather from Embark Veterinary • @krishna Yerramsetty from Infinome Biosciences, and • @Jason Daniel from Zymergen In addition to our own @Ketan (kumare3) and @Niels Bantilan. Hear their stories, followed by a Q&A. 7/26 at 9am PT. https://addevent.com/event/rT7842495

Hello, I have a super basic question! I’d like to run a local flyte cluster to test with, which I thought I would do by running

flyte start --config=flyte_local_k3d.yaml

, (after having run

make compile

already) for example when using k3d. Even though my

.k3d/kubeconfig-flyte.yaml

config file works with k3d directly, when I try to run the above command I always get an error:

{"json":{"src":"start.go:113"},"level":"error","msg":"Failed to create controller manager. error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"} {"json":{"src":"start.go:164"},"level":"panic","msg":"Failed to start Propeller, err: error building Kubernetes Clientset: Error building kubeconfig: error loading config file \"/Users/ailinyu/.k3d/kubeconfig-flyte.yaml\": no kind \"Simple\" is registered for version \"<http://k3d.io/v1alpha3\|k3d.io/v1alpha3\>" in scheme \"pkg/runtime/scheme.go:100\"","ts":"2022-07-25T17:17:20-07:00"}

Am I using this incorrectly or do I need to have something else in my k3d yaml config perhaps?

Hi, I was surprised that my long running interruptible training task was scheduled onto a non-spot instance after a failure (probably due to preemption). Thinking about it, it makes sense as default behavior, and is also documented:

If your task gets preempted, Flyte will retry your task on a non-spot (regular) instance. This retry will not count towards a retry that a user sets.

Still I'd like to configure Flyte to try one more time on a spot instance. There's a interruptible-failure-threshold ("number of failures for a node to be still considered interruptible"). Is this the right config to tweak that behavior?

thanks for that great panel discussion on flyte in biotech! @Martin Stein @Ketan (kumare3) @Niels Bantilan @Sandra Youssef and for the great insights from my fellow biotech engineers/scientists: @Calvin Leather @Jason Daniel @Kenny Workman @krishna Yerramsetty for anyone that is considering flyte for biotech (or anything for that matter), please feel free to reach out! looking forward to growing this community!

Hi Flyers, The recording of @Ketan (kumare3)’s podcast with the MLOPs community is out!

Why You Need More Than Airflow▾

Learn why we need a fabric for ML orchestration, and how to build that layer correctly. https://mlops.community/watch/why-you-need-more-than-airflow_qw3XkiPuPJFeQ9/

Multi-cluster flyte permissions question 🧵

Hi, everyone. Quick question: is the

--force

flag on

flytectl register files

working? When I use it, the CLI logs "AlreadyExists", which makes it look like it didn't actually rewrite anything. Is this normal? If so, shouldn't the logs print another message, such as "Overwritten" or something? I'm using FlyteCTL 0.5.6, by the way.

Hi, All! I would like to check which users run executions in Flyte remote server, but I guess Flyte does not provide this, am I right? I cannot see any logs who executes workflow from kubernetes. (my MLOps team is about to release Flyte to all research engineer of my company, so I want to check this point)

Newcomer question. I was wondering if anyone had tips on how to configure flyte to access GPUs. I see this document: https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/configure_use_gpus.html#sphx-glr-auto-deployment-configure-use-gpus-py But I am curious how to apply that configuration shown:

plugins:
  k8s:
    resource-tolerations:
      - <http://nvidia.com/gpu:|nvidia.com/gpu:>
        - key: "key1"
          operator: "Equal"
          value: "value1"
          effect: "NoSchedule"

I saw this thread which mentioned updating the propellor configmap. Is that the way to go? Or is there a way to update plugins via the flyte config? Thanks!

Hi, everyone. I'm trying to test out the BigQuery plugin using the example in the documentation, but when I try to convert the StructuredDataset into a DataFrame, I get this error:

Traceback (most recent call last):
      ...
      File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/types/structured/basic_dfs.py", line 63, in decode
        ctx.file_access.get_data(path, local_dir, is_multipart=True)
      File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
        raise FlyteAssertion(

Message:

    Failed to get data from  to /tmp/flyteltak17mx/local_flytekit/85e6844a8f8283f8841994437be73626 (recursive=True).

Original exception: cannot copy tree '': not a directory

User error.

No path for the resulting query file. Could it be that there was an error executing the task? Flyte says the task succeeded, but I can't check on the GCP console (I don't have the necessary permissions).

Hi folks, I have read [the excellent tutorial about distributed training](https://docs.flyte.org/projects/cookbook/en/stable/auto/case_studies/ml_training/spark_horovod/keras_spark_rossmann_estimator.html#). I'm wondering whether the sentence "'SparkBackend' uses

horovod.spark.run

to execute the distributed training function" means that flyte will launch

num_proc

spark workers on

num_proc

flyte worker? Or it just launches them on one spark worker?

If the former is the case, then how does flyte decide to launch spark workers on which set of flyte workers?

anyone here want to put their 2 cents into this HN Airflow discussion? https://news.ycombinator.com/item?id=32317558 😀

service roles can be configured at 3 levels 1. Project + domain defaults (every execution launched in this project / domain will use this SA) 2. launchplan default. (every invocation of this launchplan will use this SA) 3. Execution time override (override at invocation only for a specific execution)

somewhat related to this https://flyte-org.slack.com/archives/CNMKCU6FR/p1657727959446239 Are all notifications that come out of flyte when a workflow fails the same (from this config)? For example, I have a project that would like to customize the content of the failure notifications that get sent on a per task/workflow basis.

Hi, folks, I have encountered many pieces about project "flytectldemo" in the flytectl's documentation (e.g., https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_create_execution.html). However, I failed to find the repo and

flytectl register examples

doesn't register such a project as well. May I know where can I find that? Or it is just because the flytectl's doc is out-of-date?

Folks we need help, we are working on some core changes that will potentially improve performance of the core engine upto 100x, so a pure cached workflow will complete in less than a second, all transitions will be faster. Moreover in collaboration with Spotify we are adding a feature that makes it possible to have absolutely gigantic workflows. We would love volunteers who can help us test and benchmark it? Highly appreciated

Hello! What’s the best way to define a task config that launches a custom SageMaker hyperparameter tuning job. I am trying to create a task similar to this example. i see i can provide the

task_config

as a

HPOJob

but how do i specify which

training_task

it’s supposed to execute? and also where should i specify the parameters like

HyperparameterTuningJobConfig

, and

ParameterRanges

?

Hello #general

I need help in configuring flyte with Keycloak I have setup 2 clients, flytectl(access type public) and flytepropeller(access type confidential with client Id and secret). I am following the keycloak section in the https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html# My relevant values in the helm values file are as below secrets: adminOauthClientCredentials: # -- If enabled is true, helm will create and manage

flyte-secret-auth

and populate it with

clientSecret

. # If enabled is false, it's up to the user to create

flyte-secret-auth

as described in # https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server enabled: true clientSecret: "CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C" clientId: "flytepropeller" -- FlyteAdmin server configuration adminServer: # Refer to the server config. server: httpPort: 8088 grpcPort: 8089 security: # -- Controls whether to serve requests over SSL/TLS. secure: false # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. useAuth: true allowCors: true allowedOrigins: # Accepting all domains for Sandbox installation - "*" allowedHeaders: - "Content-Type" - "flyte-authorization" # Refer to the full structure for documentation. flyteadmin: roleNameKey: "iam.amazonaws.com/role" profilerPort: 10254 metricsScope: "flyte:" metadataStoragePrefix: - "metadata" - "admin" eventVersion: 2 testing: host: http://flyteadmin # -- Authentication configuration auth: authorizedUris: # This should point at your public http Uri. - https://flytedeployment url # This will be used by internal services in the same namespace as flyteadmin - http://flyteadmin:80 # This will be used by internal services in the same cluster but different namespaces - http://flyteadmin.nmlp.svc.cluster.local:80

# Controls app authentication config
  appAuth:
    thirdPartyConfig:
      flyteClient:
        clientId: flytectl
        redirectUri: https://<flyte deployment url>/callback
        scopes:
          - offline
          - all

  # Controls user authentication
  userAuth:
    openId:
      baseUrl: https://<keycloak production realm>/realms/nsdmlp
      scopes:
        - profile
        - openid
      clientId: flytepropeller

I have also edited the flyte-admin-secrets to have the correct client secret apiVersion: v1 data: claim_symmetric_key: cWlBYzlYWHdLN3lnaksrWUJGdStFUlRYK0RDdlk4SjVjZFJtaXBTcDBhdw== cookie_block_key: ejZPdkhrZ1crWXdib21JZHdVZ05IOGJESVp0OE5KWnNZT285KzIyRVM1dw== cookie_hash_key: Q093TUY2RTdOMW5MeFZ4Rnk1dGNzZGN5NU5aeTNWN2JTMXRPTjBLUGhQQ2JWZ3hGby9XQkVRdi84Yjk4ZEIyeEV3Zm5KYURDVzFkSjBuSGZrbS8zYVE= oidc_client_secret: CO2n3hovN0J78FqxxOVtjOtkHH5fPL9C With all the above setting, when i try to login to the flyte console via the ingress, it gives me the below message in the flyteadmin pod logs {"json":{},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2022-08-04T18:34:11Z"} {"json":{},"level":"error","msg":"Error when exchanging code oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized_client","error_description":"Invalid client secret"}","ts":"2022-08-04T18:34:12Z"} Logs from 8/5/2022, 12:02:39 AM And the chrome errors out with 403 access denied page. I have tried secret and id with and without quotes(just in case if helm was acting crazy) but it still gives the same error. Please assist. Thanks, Sujith

Hey All... qq around the runtime env of a flyte workflow. In Quickly Re-deploy Your Application, there's a note:

--fast flag will take the code from your local machine and provide it for execution without having to build the container and push it. The --force flag allows overriding your previously created package.

How does flyte decide what to push up and how does the updated code get packaged into the container without a rebuild?

hi team, the documentation states that subworkflows execute within context of the parent workflows (here). What are the implications of that?

flytekit q:

execution = flyte_remote.fetch_execution(name = execution_id)
print (execution.node_executions)

is

execution.node_executions

expected to be populated? If not, how do I get the node execution details? I'm looking for level of details similar to returned by

flytectl get execution execid --details --nodeID n0 -o yaml

Hi everyone. I'm trying to use a scheduled workflow in Flyte and I have some questions. 1. is there any way to deactivate the launch plan? 2. a new pod will be created every time the scheduler runs, is there a limit to the amount of pods created by the scheduler? Thanks

.