👋 Hey everybody, amazing project. Ive been learning and reading flyte's code. Is there any roadmap docs? so far I have only found https://github.com/orgs/flyteorg/projects/3/views/1 are there any recordings of the community syncs ? 🙂

I just updated all my Flyte components to the new release

v0.19.4

(i.e. Flyte Console

v0.52.1

, Flyte Admin

v0.6.147

, etc). When I browse to Flyte Console, instead of getting a list of my projects I get:

Failed to fetch data
invalid wire type 4 at offset 1
Retry (button)

This looks like a gRPC binary incompatibility between the latest Flyte Console and Flyte Admin. Flyte team, could you take a look?

Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?

Hi Flyers, There's a lot happening this week, so mark your calendars! For starters, join us in welcoming our guest speakers from Spotify - @Nelson Arapé & @Babis Kiosidis - who will join our Community Sync tomorrow 4/19 @9am PT, to talk about Leveraging Flyte for Frictionless Upgrades. Zoom Link Meeting Notes See you there! Flyte Team

hi team, happy Monday. We are getting

terminated in the background, manually

intermittently. No errors are reported in the POD log. Any suggestions on how to troubleshoot?

Hey folks, currently investigating Flyte as an alternative to our current setup which uses Kubeflow/Argo -- we have several pipelines with a very high degree of fan-out (on the order of ~700-1000 individual tasks). Does anyone have data points/benchmarks on how Flyte scales wrt the size of the workflow? Many of the features in the SDK (especially how flytekit works w/ native python types) look quite promising and could eliminate a lot of workarounds we've implemented to use Kubeflow.

Would people be open to a PR that adds support for specifying the shasum of a docker image in addition to the tag? Seems like a pretty minor change to

flytekit

that'd enable specifying images like `docker.io/my-image@sha256:...`/is helpful to ensure that a specific workflow is locked to an exact image. Happy to make the PR later this week.

Learn why frictionless upgrades are important at Spotify, and why Flyte was chosen as their new orchestration system, in Leveraging Flyte for Frictionless Upgrades by @Nelson Arapé.

Hi Flyers, Today's Flyte Community Sync featured: • Leveraging Flyte for Frictionless Upgrades by @Nelson Arapé. Learn why frictionless upgrades are important at Spotify, and why Flyte was chosen as their new orchestration system. • Community Updates. Meeting Notes See you all on May 3rd!

And remember to register for and attend our Meetup with the MLOps Community, happening on Wednesday 4/20 at 9am PT. MLOps EngineeringLabs: What We Learned Building End-to-end ML Applications on Flyte Featuring: • @Haytham Abuelfutuh: How Flyte Orchestrates Tasks and Workflows • @Niels Bantilan: What We Learned Building End-to-End ML Applications on Flyte, featuring lightning talks by the 3 winning teams. https://home.mlops.community/public/events/mlops-engineeringlabs-what-we-learned-building-end-to-end-ml-applications-on-flyte-2022-04-20

Hey all, my team is shifting some existing data processing code over to flyte and we are using map tasks to replace some existing parallel code, had a question about

functools.partial

and its (possible) flyte equivalent. More details in thread

I am trying the example present in documentation. Getting the following error

(flyte)  raviraja@ip-192-168-124-8  ~/Desktop/flows/flyte_flows    pyflyte run example.py:wf --n 500 --mean 42 --sigma 2

Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.

Error: No such command 'run'.

(flyte)  raviraja@ip-192-168-124-8  ~/Desktop/flows/flyte_flows    pyflyte --help
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...

  Entrypoint for all the user commands.

Options:
  -c, --config TEXT       Path to config file for use within container
  -k, --pkgs TEXT         Dot separated python packages to operate on.
                          Multiple may be specified  Please note that this
                          option will override the option specified in the
                          configuration file, or environment variable
  -i, --insecure BOOLEAN  Disable SSL when connecting to Flyte backend.
  --help                  Show this message and exit.

Commands:
  init         Create flyte-ready projects.
  local-cache  Interact with the local cache.
  package      This command produces a Flyte backend registrable package...
  serialize    This command produces protobufs for tasks and templates.

run

is not available. Am I missing something? When running normally with

python example.py

it is working fine. I have installed

flytekit==0.32.6

this version.

Flyte team - do you think there will be a Flyte

v0.19.5

release (before the 1.0.0 release)? Either way, what do you think is a timeline for the next release version? Rough estimates ok.

I tried installing all my dependenices in a docker image and provided that in remote execution

(flyte)  raviraja@ip-192-168-124-8  ~/Desktop/python-monorepo/taxonomy/taxonomy/flows/flyte_flows   flyte_demo  pyflyte run --remote --image flyte_image:latest cluster_creation_flow.py:reason_clusters_pipeline

In the UI it is coming as

https://github.com/flyteorg/flyte/issues/317 I wonder if it would be ok to make public the design for flyte copilot doc mentioned in that issue ? . Context: I am just trying to understand more about flyte and contribute if possible .

📣 Contributors of the Month 📣 Hello everyone! We’re fortunate enough to have contributors who not only strive to make Flyte better but help us reach a wider audience. On that note, we want to give shout-outs and send swag to the following list of contributors this month. • @Konstantin Slisenko has written a comprehensive article on orchestrating data pipelines at Lyft; this is the go-to article for those who’d like to know how Flyte differs from and complements Airflow. • @Guillaume Perchais and @Babis Kiosidis have published an article detailing why Spotify switched the data orchestration service from Luigi and Flo to Flyte. They’ve also extensively contributed to flytekit-java, our Java SDK. We hope to have many more of your contributions, Konstantin, Guillaume, and Babis! All contributors of the month should be available on flyte.org.

the assimilation is complete. I now type "flyte" when I mean to type "flight" 🤦‍♂️🏻

Quick question about using dynamic workflows with the imperative style

create_node()

. We have a bunch of

@tasks

which have dependencies on one another, but they are not explicit (nothing is returned or passed in). We can easily use the imperative style to define the order of the tasks in a regular

@workflow

. Does the imperative style also work for

@dynamic

workflows?

Bad news @Nastya Rusina, I bounced to Flyte Console

v0.55.0

and I'm back to

Failed to fetch data: invalid wire type 4 at offset 1

for

/console

👋 Hi everyone!

Hey team! I am configuring K8s spark operator on GCP. Do we have a replacement of the below config for GCP??

I know i'm late to the party, but congrats all on the funding round!! 🎉

Hi Flyers, Union.ai is going to PyCon Startup Row & Job Fair this week, representing the Flyte community! Meet the founders of Union.ai and check out Flyte's Quickdraw Recognition demo! Expo Hall: April 28-30th / Job Fair: May 1st https://us.pycon.org/2022/

🥁🥁🥁 And.... it's out! 🎉*Flyte v1.0*🎉 is our first major release! Flyte APIs, and tools can now be considered [stable]. Most, if not all, workflows/tasks written against FlyteKit

v1.0.0

should be expected to run against all

v1.x.x

versions. Features include, among others: • Single Binary (Fast Flyte deployment) and improved Garbage Collector [Core] • File Ignore and Script Mode (Single command to register and run workflows) [FlyteKit] • Project dashboard, Map task reporting, and Dynamic workflow rendering [FlyteConsole] Check out the full list here: https://github.com/flyteorg/flyte/releases/tag/v1.0.0 🎉🎉🎉

Hi! We are currently trying to fast-register workflows using

flytectl==0.5.21

. The workflows were serialized using

pyflyte

(from `flytekit=1.0.0`; using

fast

). The storage configuration looks like this:

... 
storage:
  type: stow
  stow:
    kind: google
    config:
      project_id: "<project>"
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
  container: "<bucket>"

Now, when running

flytectl register files tmp/workflows/* -p acorn -d development

, we see:

Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
{"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}

GOOGLE_APPLICATION_CREDENTIALS

env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0

Is there a github issue tag for issues related to user onboarding? If a current priority of flyte is to increase ecosystem size, then helping ease user onboarding would likely be important. It does not seem there are relevant tags to highlight issues related to such ends?

what’s the difference between boilerplate --> https://github.com/flyteorg/boilerplate and https://github.com/flyteorg/datacatalog/tree/master/boilerplate … would have expected https://github.com/flyteorg/boilerplate to be the place where boilerplate gets updated, and the other places to be propagated from there. the datacatalog location appears to have more recent updates, so perhaps my assumption/understanding is incorrect?

We just updated from Flyte release

v0.19.3

to new

v1.0.0

and Flyte Admin no longer wants to register workflows like:

@flytekit.workflow
def DDPTrainingWorkflow(
    train_datasets: typing.List[str] = [],
    ...
):
    patch_experiment(train_datasets)

@flytekit.task()
def patch_experiment(
    train_datasets: typing.List[str],
)

^^^^ while registering the workflow, this throws:

[2022-04-27T15:09:25Z] 	details = "Type mismatch for Parameter train_datasets in default_inputs has type collection_type:<simple:STRING > , expected simple:NONE "

clearly something doesn't like the default

[]

value of

train_datasets

against its declared type

typing.List[str]

. Ideas?