Flyte Monthly Issue # 11 is out! https://lnkd.in/gzqW_E9n Read up on the KubeRay-Flyte Integration, WhyLogs, success stories, latest talks and meet some of Flyte's contributors. Subscribe here: https://lnkd.in/gQyBmi2c

@Alex Pozimenko Please meet @Eduardo Apolinario (eapolinario) @Yee they are looking into real world use-cases for what we have been calling Signaling (Human-in-the-loop feature)…

good morning! Is there a way to disable the use of cache when choosing “Relaunch” from the Flyte console?

Our

flytepropeller

is continuously logging the following error (at a rate of ~50/second), and we have no workflows or tasks running. The last workflow was a few days ago.

{
  "json": {
    "exec_id": "f8114eedda3854878b11",
    "node": "n0/dn102/dn155",
    "ns": "dpp-default",
    "res_ver": "67801339",
    "routine": "worker-6",
    "src": "task_event_recorder.go:27",
    "wf": "dpp:default:msat.level2.workflow.level2_wf"
  },
  "level": "warning",
  "msg": "Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from SUCCEEDED to ABORTED for task execution {resource_type:TASK project:\"dpp\" domain:\"default\" name:\"msat.level2.proxy.run_splat\" version:\"dpp-b9ef0a90\"  node_id:\"n0-0-dn102-0-dn155\" execution_id:<project:\"dpp\" domain:\"default\" name:\"f8114eedda3854878b11\" >  0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!",
  "ts": "2022-08-31T16:29:14Z"
}

Is there a way to “reset” propeller and have it ignore these past errors? Seems the flyte state is in a bad state.

Hey, everyone! Is anyone else having problems starting the sandbox? A coworker of mine was trying to start it using FlyteCTL and was getting this error:

Error: Get "<https://127.0.0.1:30086/api/v1/nodes>": dial tcp 127.0.0.1:30086: connect: connection refused

When she tries to execute the Docker image

<http://cr.flyte.org/flyteorg/flyte-sandbox|cr.flyte.org/flyteorg/flyte-sandbox>

directly with

docker run

, this error happens:

...
Release "flyte-core" does not exist. Installing it now.
Error: file '/root/.cache/helm/repository/flyte-core-v1.1.0.tgz' does not appear to be a gzipped archive; got 'application/octet-stream'

I was able to reproduce it in my machine. What could be happening?

We're aware of an issue with the

flyte-core

helm chart version

1.1.0

(which is the latest release). This is impacting all scenarios, including flyte deploys and also sandbox. Fix coming up shortly.

Hey Everyone ! I'm trying to use flyte to run the diabetes classification model. I've created an EC2 instance and trying to run it. But I'm not able to change the s3 bucket at which I would like to store the meta data

Can anyone suggest where I can modify the s3 bucket name to my requirement ?

#general When trying to deploy a workflow in flyte, I am getting messages which see, to specify that I am exceeding some sort of a limit error file @[s3://flyte/metadata/propeller/brozzu-smart-ca-pipeline-development-uzxgx5b5or/n3/data/0/error.pb] is too large [17471908] bytes, max allowed [10485760] bytes" Does this mean that there is a max 10Mb size allowed for serialized uploads to flyte? Please let me know if there a limit specification for this and if so, where can I customize the same.

Hi Flyers, Listen to the story of the Schibsted group as they establish a machine learning team, define infrastructure requirements and ML workflows, evaluate and adopt Flyte, and share some of their learnings. Many thanks to @Paul Beskow, @Mücahit, @Björn Schiffler, @Yini Gao and @Oleg Ievtushok!

#general I am trying to run a flyte workflow from the console and I am getting the below error {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","routine":"worker-2","src":"handler.go:168"},"level":"info","msg":"Processing Workflow.","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"executor.go:364","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"info","msg":"Handling Workflow [an4zmzms9fccqwzzzhk7], id: [project:\"samuel-s3-data\" domain:\"development\" name:\"an4zmzms9fccqwzzzhk7\" ], p [Ready]","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"executor.go:123","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"info","msg":"Setting the MetadataDir for StartNode [s3://flytedata/metadata/propeller/samuel-s3-data-development-an4zmzms9fccqwzzzhk7/start-node/data]","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"executor.go:270","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"debug","msg":"Transitioning/Recording event for workflow state transition [Ready] -\u003e [Running]","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"admin_eventsink.go:44","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"debug","msg":"AdminEventSink received a new event execution_id\u003cproject\"samuel-s3-data\" domain:\"development\" name:\"an4zmzms9fccqwzzzhk7\" \u003e producer_id:\"propeller\" phase:RUNNING occurred_at\u003cseconds1662094333 nanos:794696270 \u003e ","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"workflow_event_recorder.go:69","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id\u003cproject\"samuel-s3-data\" domain:\"development\" name:\"an4zmzms9fccqwzzzhk7\" \u003e producer_id:\"propeller\" phase:RUNNING occurred_at\u003cseconds1662094333 nanos:794696270 \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]","ts":"2022-09-02T045213Z"} {"json":{"exec_id":"an4zmzms9fccqwzzzhk7","ns":"samuel-s3-data-development","res_ver":"39913921","routine":"worker-2","src":"executor.go:351","wf":"samuel-s3-datadevelopments3_data.my_wokflow.my_test_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]]","ts":"2022-09-02T045213Z"} I am able to run flytectl commands from console using clientsecret file correctly as well as login to the flyte console properly. However the flytepropeller pod seems to not pick the correct authentication method. Please assist me in setting the correct authentication parameters in the flytepropeller pod configurations.

Team, we configured own domain name and able to access the flyte console after running workflow. jobs are getting succeed but unable to view the pod logs from the console. link is pointing to http://localhost:30082 like below but our flyte url is https://dev-flyte.example.com. please let me know anything need to be updated. also we dont see any logs are pushed to cloudwatch after completion of job. http://localhost:30082/#!/log/cloudops-max-flyte-demo-development/f82ed701fa1f04c489[…]0-0-driver/pod?namespace=cloudops-max-flyte-demo-development

Hi everyone! I want to use Flyte for my use case, which is a pipeline of tasks that run on GPU with torch tensors on GPU. Can Flyte pass GPU tensors between tasks in a workflow? Can this be done without copies between the tasks? How can I define concurrency of tasks? I saw this is possible but no documentation on how to do it. Thank you so much in advance!

Hi, I'm sorry if this has been asked before, I've been searching for the answers but couldn't find it. In my current setup, I have a requirement to have flyte write all temporary files to a nondefault location. I was able to make it write some data (directories named like flyte+bunchofletters) to what I specify with the env variable TEMPDIR (used by pythons tempfile library) but flyte also writes some other stuff to /tmp/flyte. Can I configure also where this other stuff is written? Thanks for the help

Hi Flyers, Curious about what Flyte has been up to? Want a sneak peek into upcoming improvements and updates? Join the Flyte Community Sync tomorrow 9/6 at 9am PT for Roadmap Updates. Also, hear guest speaker @Fabio Grätz introduce his talk at the upcoming Linux Foundation Open Source Summit in Europe this month, Building Robust ML Production Systems Using OSS Tools for Continuous Delivery for ML (CD4ML). Calendar Invite and Zoom Link

Hi everyone, I am trying to update the node group instance type through env.yaml while deploying with opta. Currently it's at default t3.medium. However, adding node_instance_type with the desired instance type seems to have no effect. I am adding this under - type: k8s-cluster. Do I need to create a separate node group using opta env to achieve this?

#general I am trying to pass a tensorflow h5 model file between tasks but seems to have hit an issue. Is there any example for passing h5 files between tasks. [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes. [adpz8w6l67zrzfglb9m9-n4-0] terminated with exit code (137). Reason [Error]. Message: use python type that flyte support."} {"asctime": "2022-09-06 083946,752", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'keras.engine.training.Model'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."} Different versions of python message is also confusing since all of my tasks are present in the same container and follow the same versions.

Hey! Has anyone gotten apache-beam to run on Flyte natively? I was going to explore this a bit, but I wanted to see if anyone here has thought through this.

Github issue tracking this: https://github.com/flyteorg/flyte/issues/2855 We're going to take care of this asap. The expectation is to have another flytekit beta version by tomorrow (9/7).

For scheduling, it is done in UTC timezone, but is there a way to change just the UI so the console displays a local timezone?

#general One of my tasks is using mlflow logging facilities and when that tasks runs in the flyte pod, I am getting the below error mlflow.utils.autologging_utils: Encountered unexpected error during autologging: Unable to locate credentials This is because the flyte pod will not have the mlflow env variables pertaining to the mlflow environment like artefact store, s3 endpoint. My query is how to provide env variables to the flyte task pod which is triggered by flyte for each task

Hi Flyers, Tune in to the latest on , the upcoming Machine Learning Hangout featuring Flyte & Ray, and hear MLOps Lead @Fabio Grätz present an intro on and a typical software adoption journey, presented in depth at the Open Source Summit in Europe next week. Also check out a Flyte Ecosystem project .

could you share your docker file?

hi all 👋, I just wanted to ping the community to ask a quick question 🤔: Say you have workflow that uses a trained model to generate predictions on a scheduled launchplan. The question is, how do you typically want to get features for that prediction? I.e. when the scheduled workflow kicks off, where are you reading those features from? Do you need the kick-off time as a parameter to fetching data from, say, an s3 bucket or DB?

Question about BQ plugin: (ref doc https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/gcp/bigquery/index.html) Will the flyte task also include monitoring/checking if results for the query is complete? Most of our queries does take a while to execute, so wondering how will the pods behave in this case?

Question, does flyte support continues training and hyper parameter optimisation?

Hello! We’re running Flyte on AWS EKS (k8s). Due to the security concerns we have a database which resides in VPC separate from the EKS. We would like to access that db from one task without giving access to the whole k8s cluster to that VPC. Do you know good strategies to achieve that? So far the ideas we’ve came up with were around using some other AWS service like Fargate or Glue to run the task (as just that task can be run within that specific VPC) and trigger it from Flyte. That feels overly complex however.

@Prafulla Mahindrakar, could you help?

Newly formed PyTorch Foundation just announced: https://ai.facebook.com/blog/pytorch-foundation/

Folks, Please thank @Bernhard Stadlbauer for adding Pachama to the list of users of Flyte. I know there are ton of more users, who have not added themselves. So if you are using Flyte, adding your name will really help the community. Thank you so much Pachama. https://github.com/flyteorg/flyte/pull/2869