• Jake Neyer

    Jake Neyer

    4 months ago
    Hey all! As promised, thanks largely to @Matthew Griffin, we are dropping our Flyte VSCode extension for anyone who is interestedhttps://github.com/Striveworks/Flyte-Wingman We would love feedback and contributions!
    Jake Neyer
    Niels Bantilan
    +2
    7 replies
    Copy to Clipboard
  • George Odette

    George Odette

    4 months ago
    Hi all, not flyte related. But has anyone ever experienced this while working with docker? Trying to upload a workflow
    George Odette
    1 replies
    Copy to Clipboard
  • Samhita Alla

    Samhita Alla

    3 months ago
    📣 Contributors of the Month 📣 Hi everyone! For the month of May, we want to give shout-outs and send swag to the following list of contributors who are sweating it out to make Flyte even better! @eugene jahn contributed to FlyteConsole, our UI. He’s now part of our team, so you can expect consistent contributions from him. • @Nick Müller (MorpheusXAUT) worked on a couple of features and fixes, including standardizing flyteidl docs generation, improving the local set-up for contributors, and adding an interruptible override to launch forms. We hope to have many more of your contributions, Eugene and Nick! All contributors of the month should be available on flyte.org.
  • Sandra Youssef

    Sandra Youssef

    3 months ago
    Hi Flyers, Learn how MLOps is actually a combination of Machine Learning, DevOps, and Data Engineering, and how Flyte can help solve MLOps challenges in this blog by Union.ai's @Samhita Alla in cooperation with the MLOps Community.https://mlops.community/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering/
  • Sandra Youssef

    Sandra Youssef

    3 months ago
    Learn how we are growing the Flyte ecosystem, the latest releases and improvements, upcoming conference talks, and soon-to-be-launched UnionML! All in Flyte Monthly Issue #8. Out now!https://www.getrevue.co/profile/flyte/issues/flyte-monthly-issue-8-1157760 Subscribe here: https://www.getrevue.co/profile/flyte
  • Sandra Youssef

    Sandra Youssef

    3 months ago
    Hi Flyers, Join our Community Sync tomorrow 5/31, featuring: • Flyte Contributor of the Month guest appearances • Flyte blog writer guest appearance • Flyte ML projects - UnionML and Flyte-Wingman VS Code Extension - short talks • FlyteConsole UI updates with @Jason Porter, and • MethaneSAT guest speaker, @Nicholas LoFaso presenting "Transforming Satellite Data with Flyte @ MethaneSAT." Not to be missed! See you tomorrow 9am PT on Zoom! Flyte Teamhttps://addevent.com/event/EA7823958
  • e

    Eugene Cha

    3 months ago
    we're trying to run the caching.py example to see how the caching works, but it appears to only work sometimes. we increased the sleep time to 50 seconds
    def hash_pandas_dataframe(df: pandas.DataFrame) -> str:
        return str(pandas.util.hash_pandas_object(df))
    
    
    @task
    def uncached_data_reading_task() -> Annotated[
        pandas.DataFrame, HashMethod(hash_pandas_dataframe)
    ]:
        return pandas.DataFrame({"column_1": [1, 2, 3]})
    
    
    @task(cache=True, cache_version="1.0")
    def cached_data_processing_task(df: pandas.DataFrame) -> pandas.DataFrame:
        time.sleep(50)
        return df * 2
    
    
    @task
    def compare_dataframes(df1: pandas.DataFrame, df2: pandas.DataFrame):
        assert df1.equals(df2)
    
    
    @workflow
    def cached_dataframe_wf():
        raw_data = uncached_data_reading_task()
    
        # We execute `cached_data_processing_task` twice, but we force those
        # two executions to happen serially to demonstrate how the second run
        # hits the cache.
        t1_node = create_node(cached_data_processing_task, df=raw_data)
        t2_node = create_node(cached_data_processing_task, df=raw_data)
        t1_node >> t2_node
    
        # Confirm that the dataframes actually match
        compare_dataframes(df1=t1_node.o0, df2=t2_node.o0)
    
    
    if __name__ == "__main__":
        df1 = cached_dataframe_wf()
        print(f"Running cached_dataframe_wf once : {df1}")
    but sometimes the caching works and sometimes it doesnt. we've tried running with pyflyte run --remote caching.py cached_dataframe_wf as well as trying the relaunch button but as you can see in the pictures it tends to not work and i'm not sure why. any ideas?
    e
    p
    +2
    26 replies
    Copy to Clipboard
  • Emirhan Karagül

    Emirhan Karagül

    3 months ago
    Hi everybody, We have been using flyte schedules for a while now. Yesterday something mysterious happened. A scheduled launchplan got executed 15 hours earlier. Usually the execution used to start around 3 seconds after 5:00 AM UTC. Anybody has an idea on how I can debug this and see what happened? (I hope it happened due to some cosmic bit flipping 😄) Thank you!
    ~ flytectl get execution -p default -d production f38f2b53cfda08ccb000 -o yaml
    closure:
      createdAt: "2022-05-31T14:03:10.025255897Z"
      duration: 501.772263113s
      outputs:
        uri: gs://<our-flyte-store>/metadata/propeller/default-production-f38f2b53cfda08ccb000/end-node/data/0/outputs.pb
      phase: SUCCEEDED
      startedAt: "2022-05-31T14:03:15.128550714Z"
      updatedAt: "2022-05-31T14:11:36.900814113Z"
      workflowId:
        domain: production
        name: flyte_workflows.collaborative_filtering.workflow.pipeline
        project: default
        resourceType: WORKFLOW
        version: 0.2.2
    id:
      domain: production
      name: f38f2b53cfda08ccb000
      project: default
    spec:
      launchPlan:
        domain: production
        name: hydra_workflow_cfg_flyte_workflows.collaborative_filtering.workflow_0
        project: default
        resourceType: LAUNCH_PLAN
        version: 0.2.2
      metadata:
        mode: SCHEDULED
        scheduledAt: "2022-06-01T05:00:00Z"
        systemMetadata: {}
    Emirhan Karagül
    p
    7 replies
    Copy to Clipboard
  • Robin Kahlow

    Robin Kahlow

    3 months ago
    Is it not possible to have default arguments for lists on workflows? eg. when i try to
    pyflyte run
    on
    @workflow
    def wf(
        total_samples: List[int] = [16, 32, 64, 256],
    ):
    I get
    TypeError: the JSON object must be str, bytes or bytearray, not list
    but without specifying a default it does work (and i can pass it to pyflyte as a string json list)
    Robin Kahlow
    Samhita Alla
    4 replies
    Copy to Clipboard
  • Robin Kahlow

    Robin Kahlow

    3 months ago
    Trying to use GPUs, I added a tolerations section as described here https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_use_gpus.html (and in a previous comment where it was clarified where to apply this https://flyte-org.slack.com/archives/CNMKCU6FR/p1651591056890689?thread_ts=1651584781.772139&amp;cid=CNMKCU6FR), ie.
    # -- Kubernetes specific Flyte configuration
      k8s:
        plugins:
          # -- Configuration section for all K8s specific plugins [Configuration structure](<https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config>)
          k8s:
            default-env-vars: []
            #  DEFAULT_ENV_VAR: VALUE
            default-cpus: 100m
            default-memory: 100Mi
    
            resource-tolerations:
              - <http://nvidia.com/gpu|nvidia.com/gpu>:
                - key: "key1"
                  operator: "Equal"
                  value: "value1"
                  effect: "NoSchedule"
    and I applied that with helm and also tried restarting the Flyte pods (kubectl rollout restart deploy), but the pods that get started by Flyte workflows don't get these tolerations (although they do get a default nvidia.com/gpu "exists" toleration regardless of my addition above). Anything I'm doing wrong?
    Robin Kahlow
    k
    20 replies
    Copy to Clipboard