Hi all, not flyte related. But has anyone ever experienced this while working with docker? Trying to upload a workflow

📣 Contributors of the Month 📣 Hi everyone! For the month of May, we want to give shout-outs and send swag to the following list of contributors who are sweating it out to make Flyte even better! • @eugene jahn contributed to FlyteConsole, our UI. He’s now part of our team, so you can expect consistent contributions from him. • @Nick Müller (MorpheusXAUT) worked on a couple of features and fixes, including standardizing flyteidl docs generation, improving the local set-up for contributors, and adding an interruptible override to launch forms. We hope to have many more of your contributions, Eugene and Nick! All contributors of the month should be available on flyte.org.

Hi Flyers, Learn how MLOps is actually a combination of Machine Learning, DevOps, and Data Engineering, and how Flyte can help solve MLOps challenges in this blog by Union.ai's @Samhita Alla in cooperation with the MLOps Community. https://mlops.community/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering/

Learn how we are growing the Flyte ecosystem, the latest releases and improvements, upcoming conference talks, and soon-to-be-launched UnionML! All in Flyte Monthly Issue #8. Out now! https://www.getrevue.co/profile/flyte/issues/flyte-monthly-issue-8-1157760 Subscribe here: https://www.getrevue.co/profile/flyte

Hi Flyers, Join our Community Sync tomorrow 5/31, featuring: • Flyte Contributor of the Month guest appearances • Flyte blog writer guest appearance • Flyte ML projects - UnionML and Flyte-Wingman VS Code Extension - short talks • FlyteConsole UI updates with @Jason Porter, and • MethaneSAT guest speaker, @Nicholas LoFaso presenting "Transforming Satellite Data with Flyte @ MethaneSAT." Not to be missed! See you tomorrow 9am PT on Zoom! Flyte Team https://addevent.com/event/EA7823958

we're trying to run the caching.py example to see how the caching works, but it appears to only work sometimes. we increased the sleep time to 50 seconds

def hash_pandas_dataframe(df: pandas.DataFrame) -> str:
    return str(pandas.util.hash_pandas_object(df))


@task
def uncached_data_reading_task() -> Annotated[
    pandas.DataFrame, HashMethod(hash_pandas_dataframe)
]:
    return pandas.DataFrame({"column_1": [1, 2, 3]})


@task(cache=True, cache_version="1.0")
def cached_data_processing_task(df: pandas.DataFrame) -> pandas.DataFrame:
    time.sleep(50)
    return df * 2


@task
def compare_dataframes(df1: pandas.DataFrame, df2: pandas.DataFrame):
    assert df1.equals(df2)


@workflow
def cached_dataframe_wf():
    raw_data = uncached_data_reading_task()

    # We execute `cached_data_processing_task` twice, but we force those
    # two executions to happen serially to demonstrate how the second run
    # hits the cache.
    t1_node = create_node(cached_data_processing_task, df=raw_data)
    t2_node = create_node(cached_data_processing_task, df=raw_data)
    t1_node >> t2_node

    # Confirm that the dataframes actually match
    compare_dataframes(df1=t1_node.o0, df2=t2_node.o0)


if __name__ == "__main__":
    df1 = cached_dataframe_wf()
    print(f"Running cached_dataframe_wf once : {df1}")

but sometimes the caching works and sometimes it doesnt. we've tried running with pyflyte run --remote caching.py cached_dataframe_wf as well as trying the relaunch button but as you can see in the pictures it tends to not work and i'm not sure why. any ideas?

Hi everybody, We have been using flyte schedules for a while now. Yesterday something mysterious happened. A scheduled launchplan got executed 15 hours earlier. Usually the execution used to start around 3 seconds after 5:00 AM UTC. Anybody has an idea on how I can debug this and see what happened? (I hope it happened due to some cosmic bit flipping 😄) Thank you!

~ flytectl get execution -p default -d production f38f2b53cfda08ccb000 -o yaml
closure:
  createdAt: "2022-05-31T14:03:10.025255897Z"
  duration: 501.772263113s
  outputs:
    uri: gs://<our-flyte-store>/metadata/propeller/default-production-f38f2b53cfda08ccb000/end-node/data/0/outputs.pb
  phase: SUCCEEDED
  startedAt: "2022-05-31T14:03:15.128550714Z"
  updatedAt: "2022-05-31T14:11:36.900814113Z"
  workflowId:
    domain: production
    name: flyte_workflows.collaborative_filtering.workflow.pipeline
    project: default
    resourceType: WORKFLOW
    version: 0.2.2
id:
  domain: production
  name: f38f2b53cfda08ccb000
  project: default
spec:
  launchPlan:
    domain: production
    name: hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
    project: default
    resourceType: LAUNCH_PLAN
    version: 0.2.2
  metadata:
    mode: SCHEDULED
    scheduledAt: "2022-06-01T05:00:00Z"
    systemMetadata: {}

Is it not possible to have default arguments for lists on workflows? eg. when i try to

pyflyte run

on

@workflow
def wf(
    total_samples: List[int] = [16, 32, 64, 256],
):

I get

TypeError: the JSON object must be str, bytes or bytearray, not list

but without specifying a default it does work (and i can pass it to pyflyte as a string json list)

Trying to use GPUs, I added a tolerations section as described here https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_use_gpus.html (and in a previous comment where it was clarified where to apply this https://flyte-org.slack.com/archives/CNMKCU6FR/p1651591056890689?thread_ts=1651584781.772139&cid=CNMKCU6FR), ie.

# -- Kubernetes specific Flyte configuration
  k8s:
    plugins:
      # -- Configuration section for all K8s specific plugins [Configuration structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config>)
      k8s:
        default-env-vars: []
        #  DEFAULT_ENV_VAR: VALUE
        default-cpus: 100m
        default-memory: 100Mi

        resource-tolerations:
          - <http://nvidia.com/gpu|nvidia.com/gpu>:
            - key: "key1"
              operator: "Equal"
              value: "value1"
              effect: "NoSchedule"

and I applied that with helm and also tried restarting the Flyte pods (kubectl rollout restart deploy), but the pods that get started by Flyte workflows don't get these tolerations (although they do get a default nvidia.com/gpu "exists" toleration regardless of my addition above). Anything I'm doing wrong?

Hi Flyers, Thank you for attending this week's Flyte Community Sync. Special thanks to our contributors who made guest appearances to talk about their work with Flyte, and our guest speaker Nicholas LoFaso. The recordings are now available on YouTube: • Community Updates & Contributor guest appearances, by @Martin Stein, Union.ai. • FlyteConsole UI Updates, by @Jason Porter, Union.ai. • Transforming Satellite Data with Flyte @ MethaneSAT, by @Nicholas LoFaso, MethaneSAT. Meeting notes Join us for the next community sync on June 14th! Flyte Team

👋 Hey Flyte community, I am sketching what I learnt while doing some tiny contributions. Hopefully to pave the road for upcoming contributors. Currently sketching them in the link below. Any advices on where would be the best place for such content to live?. I was thinking of adding a

contributor_guide.rst

to

flyteorg/flyte

. https://github.com/dav009/flyte/blob/dp-contributor-fastrack/rsts/community/contributor_fast_track.md

Hi guys! Is there a way to invalidate the cache for a certain input for a task within a workflow, without increasing the cache version/invalidating the cache of other inputs?

Hi all 👋🏽 The new release of FlyteConsole v1.1.1 is now available. 🎉 More info available in #flyte-console channel.

Hey, everyone! I'm trying to execute the Flyte sandbox but this is happening. What could it be?

Deploying Flyte...
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "<https://googlecloudplatform.github.io/spark-on-k8s-operator>" chart repository
...Successfully got an update from the "<https://kubernetes.github.io/dashboard/>" chart repository
...Successfully got an update from the "<https://charts.bitnami.com/bitnami>" chart repository
Error: can't get a valid version for repositories contour. Try changing the version constraint in Chart.yaml

Hi, I am trying to assign GPU (Nvidia MIG - Multi-Instance GPU) on flyte. However, I cannot see how I assign it on each tasks in flyte (there were no mentions in flyte docs). Does anyone have a solution to it?

Is anyone in the flyte community currently using pants as their build system? In particular, curious if there are any particular considerations for running flyte workloads that are packaged as pex binaries

Hi, I am trying to use lists of FlyteFiles inside dataclasses. It seems to me, that when passing the dataclass from one task to the other, the reference to the remote source is lost. See the following example:

from dataclasses import dataclass
from flytekit import task, workflow
from typing import List

from dataclasses_json import dataclass_json
from flytekit.types.file import FlyteFile


@dataclass_json
@dataclass
class InputsContainer:
    files: List[FlyteFile]


@task
def task1(inputs: List[FlyteFile]) -> InputsContainer:
    print("TASK1 remote source: ", inputs[0].remote_source)
    return InputsContainer(files=inputs)


@task
def task2(inputs: InputsContainer) -> None:
    print("TASK2 remote source: ", inputs.files[0].remote_source)


@workflow
def main_workflow(inputs: List[FlyteFile]) -> None:
    task1_outputs = task1(inputs=inputs)
    task2(inputs=task1_outputs)


if __name__ == '__main__':
    file_path = FlyteFile("<s3://test-bucket/test.json>")
    main_workflow(inputs=[file_path])

The output generated is:

TASK1 remote source:  <s3://test-bucket/test.json>

TASK2 remote source:  None

Could anyone help me out here? Thanks!

\

I have finished up this issue by using the pod example on this link. someone who wants to try MIG will define a pod spec through k8s module. I attach the sample codes through gist

📣📣📣 Union.ai is proud to announce the release of UnionML, an open-source MLOps framework built on Flyte, that can bundle Python functions into ML microservices. It is the only library that seamlessly manages both data science workflows and production lifecycle tasks. UnionML's release is being announced today at MLOps World 2022 - Toronto, where creator @Niels Bantilan will be hosting a demo booth this afternoon and presenting a talk tomorrow, 6/10 at 1:45pm EDT, titled UnionML: a Microframework for Building Machine Learning Applications. Check it out: • Press release: Union.ai releases UnionML for seamless creation of web-native machine learning applications • Website: https://www.union.ai/unionml • Demo • Blog post: Union.ai releases UnionML for seamless creation of web-native machine learning applications • Release notes: https://github.com/unionai-oss/unionml/releases/tag/v0.1.0 • Star it on GitHub!

Also, join #unionml for more information and latest updates!

Cross posting from https://flyte-org.slack.com/archives/C01P3B761A6/p1654537014728059 to see if anybody here might know 🙂 . I tried restarting the

flytepropeller

deployment and it’s hard to tell if the system is picking up the changes:

$ kubectl logs -f deployment/flytepropeller -nflyte | grep -i logs
Found 6 pods, using pod/flytepropeller-785bcc6f6d-sr8bl
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.logs] updated. No update handler registered.","ts":"2022-06-08T16:32:15Z"}
{"json":{"src":"viper.go:396"},"level":"debug","msg":"Config section [plugins.logs] hasn't changed.","ts":"2022-06-10T22:36:33Z"}

From looking at the plugins — it seems like adding that config should just automatically work

Startrd working on a cookiecutter template here https://github.com/RobinKa/flyte-template, I wanted to make it easy to build and execute workflows for new people, so after creating a project with it all you have to do (besides having docker, flytectl set up) is install the dev dependencies and run

python commands.py build-execute.py --version v1

👋 Hi everyone! Is this the correct channel to post questions about UnionML?

Hi Flyers, Join the Flyte Community Sync tomorrow 6/14 for: • Community member appearances • The all-new FlyteDecks by @Eduardo Apolinario (eapolinario) • Guest speaker, @Matthew Griffin, presenting "*Pterodactyl, Javascript SDK for Flyte*." 9am PT Calendar Invite & Zoom Link See you there! Flyte Team

Hi @Ketan (kumare3) and anyone else, is there a flyte version of this functionality?

Hello guys, Is there a detailed document on Flyte Map tasks. I just see a very superficial page in the website.

This message was deleted.

Hey, is there documentation for the new pod templates feature?

Anyone have any experience adding caching to

map_task

? I've tried by adding the caching stuff to the

@task

decorator and in the "mapification" like this:

map_task(my_task)(input=input).with_overrides(cache=True, cache_version="1.0")

but I'm not seeing in the flyte console the little swirly that indicates the task either wrote to or read from the cache. Is caching a map task supported?