Hi @Evan Sadler @Kevin Su I'm trying to follow this guide and run a Flyte workflow on Databricks. Configured the dbx plugin based on this guide: https://docs.flyte.org/en/latest/deployment/plugins/webapi/databricks.html#deployment-plugin-setup-webapi-databricks I'm trying to take the easy way out first and run the workflow on an existing cluster. Installed

pyflyte

on the cluster and triggered the run like this:

pyflyte run -remote dbx_example_existing_cluster.py my_databricks_job

The Databricks job is started, but failing with this error:

TypeError: loader must define exec_module()

Hi all I am investigating Flyte and are stuck on a problem. I uploaded a file to the minio s3 hosted by flytectl demo. I then supply the uri s3://my-s3-bucket/dataset/file.asf as Flytefile("s3://my-s3-bucket/dataset/file.asf") I get: Protocol not found Did you mean file:s3://my-s3-bucket/dataset/file.asf? I am new to S3 so maybe I am doing something wrong here Edit added task snippet:

@task(cache=True, cache_version="2.0")
def process_video() -> FlyteFile:
    """
    Run clip_extract on video
    """
    input_path = FlyteFile("<s3://my-s3-bucket/dataset/test.asf>")
    output_file = FlyteFile("output.mjpg")
    command = ['ffmpeg', '-i', str(input_path), '-c:v', 'mjpeg', '-q:v', '3', '-an', str(output_file)]
    subprocess.check_call(command)
    return output_file

Is there a way to access a property of the workflow arguments in the workflow code? When I do something like this, it throws

TypeError: 'Promise' object is not subscriptable

since flyte creates promises during packaging/deployment:

@workflow
def wf(wf_args) -> int:
    print(wf_args["x"])
    ...

S3 Storage: Separate Backends for Metadata and for User Data? Cheers everyone, We have the following problem: Our Workflow & Task authors want to read & write files stored in a S3 backend, ideally via the Flyte Files. However, we don't want to have Flyte Metadata in the same endpoint. Can I configure the Flyte Pods executing the Tasks to have access to the "user S3 backend", without Flyte trying to store all its metadata there as well? Details: Locally on their laptops, we configured the

~/.flyte/config.yaml

, providing user-backend values for

storage.connection.endpoint, access-key, secret-key

etc, and executing the workflow via

python myworkflow.py

works like a charm -

FlyteFile("<s3://my-bucket/my-file.csv>").download()

accesses the user-backend configured in the

config.yaml

and downloads the file. Now regarding the Flyte deployment, I'm a bit lost which component (Admin, Propeller, ...) to configure so that the Pods have access to the s3-user-backend, without automatically trying to write their metadata (e.g.,

<s3://flyte/metadata/propeller/myproject-development-ffba3b040e85d4801a9c/n1/data/0>

)?

Hi Team, Getting this issue in the FlyteConsole. It was working fine and I have just redeployed the helm. What could be the reason?

Hi, I am trying to use kuberay operator. I have created kuberay-operator and ray-cluster using helm chart and it is running in a new namespace. When I am trying to run an example code mentioned in the blog here , getting this below error.

File "/home/kkanagar/kkr/kuberay_test.py", line 15, in <module>
    def ray_task() -> list[int]:
TypeError: 'type' object is not subscriptable

Changing this list[int] to any is working fine in local but when trying to register to remote flyte. Need your help, please

Hi All, I have a script with a few tasks and the workflow defined. Certain information is stored in a config file which I am importing into this script. But when I run the workflow, it is giving a module not found error. What should I be doing to import other scripts ?

Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:

<http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>

<http://prometheus.io/port=15020|prometheus.io/port=15020>

<http://prometheus.io/scrape=true|prometheus.io/scrape=true>

And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?

Question, does flyte support continues training and hyper parameter optimisation?

Hey! Has anyone gotten apache-beam to run on Flyte natively? I was going to explore this a bit, but I wanted to see if anyone here has thought through this.