Hi Flyers, If you didn't get a chance to attend our meetup with the MLOps community last week, you can now watch the just-released recording!

Hey Flyters, bit of a self-promotion here but I produce a podcast about open-source software called Contributor, and we just featured Flyte founder Ketan Umare on today's episode: https://www.contributor.fyi/flyte

Thanks for checking it out!

@katrina there were some changes to the auth in Flyte Admin a couple weeks ago that are now in

v1.0.0

that have partially broken the

kubernetes_service_account

, i.e. some of our workflows are executing with the declared service account and some are just executing with

default

. I'm still working on debugging it.

Hi there. I have Docker containers deployed on GKE. They got deployed and starts running at once. Output of one is based as input of next incoming container. Do we have such type of support in flyte so that it allows first one to finish and then let the next in the row to perform its job?

We started working on adding a Flyte source extension to Datahub for tracking metadata and lineage about our Workflows. Is anyone else maybe already working on something similar?

Hi What should be the output of the command (from the GCP(GKE) setup docs) which verifies the connection of db with GKE? I am getting attached outcome. For both Public and Private SQL IP. Whereas I am able to connect to psql with my local system through Public IP

Can someone share a working example for

Multiple Image Based Implementation in Flyte

. I can't see the implementation in docs as it is left as todo and I'm

new to Flyte

.

Hi Flyers, Flyte Monthly - Issue #7 is finally here! Featuring: • Flyte 1.0 Phoenix • Quickdraw Recognition demo at PyCon StartUp Row this week • The story of Flyte • MLOps Community meetup • Success Stories, and more! https://www.getrevue.co/profile/flyte/issues/flyte-monthly-issue-7-1087090 Subscribe here: https://t.co/3gycgieIgv

How to

map volumes

through flyte from a directory inside VM to a directory inside Container and vice versa.(Bind mount)

Raw containers cannot be run locally at the moment.

What does this statement mean? So, even if I have local images, I am unable to implement Flyte on them?

Hi Flyers, Join us tomorrow 5/3 at the Flyte Community Sync, for updates and learn about the all-new UnionML! Calendar invite and Zoom link: https://addevent.com/event/EA7823958 See you then! Flyte Team

hi team, is it possible to get recent failed executions across all projects using flyte admin api?

Hi! If I define a workflow with a dataclass/struct as input, along the lines of:

@dataclass_json
@dataclass
class Foo:
    bar: int
    baz: float
    

@workflow
def my_wf(foo: Foo):
    ...

Is it possible to introspect the fields of the

STRUCT

or is this checked at runtime?

It seems that after this change our flyte services have excessive logging related to gorm, logging queries and formatting is ignored etc. more in the 🧵

I just tried upgrading to

flyteadmin:1.0.1

for the stow fix https://github.com/flyteorg/flyteadmin/pull/416 but now flyteconsole fails with a 500. "Request failed with status code 500 header key "userinfo" contains value with non-printable ASCII characters"

Hey, looking into configuring Flyte to access GPUs. Where do I specify the plugins in the Helm values for the Flytepropeller? Do they go here, or should I enter them elsewhere?

It's finally out - Type-Safe Data Processing and Machine Learning Pipelines with Flyte and Pandera! From Data Council Austin by @Niels Bantilan

Hi Flyers, Check out this quick intro to UnionML, an app that can be used to scale training jobs remotely and then connect to Flyte, presented by @Niels Bantilan. Also watch this week's community updates by @katrina. Mark your calendars for the next community sync on May 17th, which will feature guest speaker @jeev from Freenome. Details to come. See you all then! Flyte Team

Hey, everyone. I updated flytekit and pyflyte to test the v1.0.0 and the new serialization process broke my code. The problem is that I have a decorator called

@task_setup

that, as the name suggests, executes some startup/teardown code before and after each task. This decorator is defined inside a submodule

tasks.utils.flyte

. My tasks are inside the

tasks

module, separated by different modules. When I packaged the tasks before, their names were `tasks.<module>.<task_name>`; now, their names are

tasks.utils.flyte.<task_name>

. Basically, the path for the decorator is being used as the "prefix" for the task. This is not ideal and the old method makes way more sense... What should I do?

Also, is flytectl doing something special when starting the demo? Because I'm trying to start it using Docker and it takes too long... My Docker Compose file, which worked perfectly for the old

flyte-sandbox

(I just altered the ports):

services:
  sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565|cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565>
    container_name: flyte-sandbox
    privileged: true
    ports:
      - "30080:30080"         # Console at <http://localhost:30080/console>
      - "30081:30081"
      - "30082:30082"
      - "30084:30084"
      - "30086:30086"
      - "30088:30088"
      - "30089:30089"

https://github.com/flyteorg/flyte/issues/2433 <- can someone help with this? Getting failure notices makes me not want to file issues 😕

Hi! Do I need a specific version of helm/kubectl to install the helm charts from the flyte repo? I keep getting this error

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.securityContext): unknown field "fsGroupChangePolicy" in io.k8s.api.core.v1.PodSecurityContext

Hi Flyers, Learn how to create event-driven applications with AWS Lambda and Flyte in this blog, which demonstrates this using a fun and simple Neural Style Transfer model. Written by Union.ai's @Samhita Alla. https://lnkd.in/gFXatR38

As promised, we're publishing a security advisory for a security vulnerability in flyteconsole. We recommend users to upgrade to flyteconsole v0.52.0 or greater ASAP (which corresponds to Flyte v0.19.4). As mentioned earlier, we yanked all affected versions from the official image repository. We've also reached out to affected folks individually and believe that all impacted deployments have been patched. For more details about the exploit, please visit https://github.com/flyteorg/flyteconsole/security/advisories/GHSA-www6-hf2v-v9m9.

Hey, everyone! I'm kinda confused on how to use custom images on a sandbox with Flyte 1.x.x. At first I thought the sandbox would pull the image from my local repository, but this doesn't seem to be the case, since I'm receiving the error

containers with unready status: [ash8dlh4m294mnd9psbs-f62iya2y-0]|Back-off pulling image "flyte-example:1.0"

when trying to execute a workflow, even though I have the flyte-example:1.0 image on my local repo. But, at the same time, it doesn't seem like I should have to build the image inside the sandbox container, right? The container from

flytectl demo start

doesn't even mount the working directory into

/root

. I can't find anything in the documentation that says how to do this...

Hello guys! I need some help. We are doing some testing with flyte in the AWS environment with ALB. But I'm having trouble trying to run the example code:

import typing
import pandas as pd
import numpy as np

from flytekit import task, workflow

@task
def generate_normal_df(n:int, mean: float, sigma: float) -> pd.DataFrame:
    return pd.DataFrame({"numbers": np.random.normal(mean, sigma,size=n)})

@task
def compute_stats(df: pd.DataFrame) -> typing.Tuple[float, float]:
    return float(df["numbers"].mean()), float(df["numbers"].std())

@workflow
def wf(n: int = 200, mean: float = 0.0, sigma: float = 1.0) -> typing.Tuple[float, float]:
    return compute_stats(df=generate_normal_df(n=n, mean=mean, sigma=sigma))

When I run the code using the local cluster using flytectl demo start (k3s) it works. But in production environment using EKS and ALB does not work.

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2                                                                                     SIGINT(2) ↵  10055  17:06:22 
{"asctime": "2022-05-06 17:06:31,287", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867591.287595583\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867591.287594125\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-05-06 17:06:32,274", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867592.274550785\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867592.274550285\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flytekit/clis/sdk_in_container/run.py", line 471, in _run
    wf = remote.register_script(
  File "/usr/local/lib/python3.8/dist-packages/flytekit/remote/remote.py", line 536, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/usr/local/lib/python3.8/dist-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1651867592.675517306","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1651867592.675516568","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"

Do you have any ideas? I'm using helm with values-eks.yaml

I was looking at best practices with CI/CD & Flyte in your documentation a few weeks back but I can’t seem to find it anymore. Is there such a document available?

Is there a way to configure the storage location/prefix for code when using the new fast registration without bucket credentials? It used to be stored under

bucket-name/fast/

before (not sure if I had configured it before, I can't find any config related to that). Now it's just stored in random paths under the bucket root.

hi team, we're having issues with metric collection on the recent flyteadmin. Telegraf starts throwing lots of these after just a day or two of uptime and some metrics disappear from the dashboard:

Error in plugin [inputs.prometheus]: error reading body: net/http: request canceled (Client.Timeout exceeded while reading body)

original settings were:

interval = "10s"
response_timeout = "3s"

I changed these to 30s and 15s and it seems to be ok for now. Do you have a recommendation on that these should be set to?