Have anyone try the https://docs.flyte.org/projects/cookbook/en/latest/bioinformatics_examples.html Bioinformatics Tutorial, I was able to run it locally but when I run it on Flyte cluster it failed with this message:

Traceback (most recent call last):

      File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 499, in dispatch_execute
        raise e
      File "/usr/local/lib/python3.10/site-packages/flytekit/core/base_task.py", line 496, in dispatch_execute
        native_outputs = self.execute(**native_inputs)
      File "/usr/local/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 215, in execute
        subprocess.check_call(gen_script, shell=True)
      File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
        raise CalledProcessError(retcode, cmd)

Message:

    Command '
    mkdir -p output

    query=data/kitasatospora/k_sp_CB01950_penicillin.fasta
    db=data/kitasatospora/kitasatospora_proteins.faa
    blastout=output/AMK19_00175_blastx_kitasatospora.tab

    blastx -out $blastout -outfmt 6 -query $query -db $db >> stdout.txt 2>&1
    ' returned non-zero exit status 127.

SYSTEM ERROR! Contact platform administrators.

Any idea how to fix this?

Launch Plans launched in the UI seem to silently ignore "default_inputs" that are structs when specified in Python. So e.g.

@dataclass_json
@dataclass
class MyStruct
   foo: str
   bar: str = 'baz'

def workflow(s: MyStruct):
   pass

qux = LaunchPlan(workflow, default_inputs=dict(s=MyStruct(foo='baz', bar='qux')))

If i go to launch

workflow

with the

qux

launch plan it instead takes the default values from the struct definition. So

foo

will be unset and

bar

will be set by default as

baz

Not sure where the error is here, though. Has anyone else seen this?

hey, all. no matter which commands I try my

flytectl

consistently fails to discover where config lives (

~/.flyte/config.yaml

). the only way i've gotten it to work is by supplying

--file

arg manually – but I thought that was the point of having a default location:

> flytectl config init
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
Init flytectl config file at [/Users/kevinschaich/.flyte/config.yaml]
                                                                                                          
> flytectl config discover
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.

just me or has anyone else encountered? did I forget to set an env variable somewhere? I'm on an M1 Mac – Ventura

13.2

,

flytectl v0.6.27

Hi all, building my first more complex flyte workflow and I have a couple of questions on best practices. I have a workflow defined in

project_1

that includes a dynamic workflow. The dynamic workflow starts a launchplan defined in

project_2

(see attached sketch). This is my code structure:

|- project_1 /
|  |- pyproject.toml
|  |- project_1 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- project_2 /
|  |- pyproject.toml
|  |- project_2 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- Dockerfile
|- docker_build.sh

What I'm doing right now: • add

project_2

as a dependency to project one • build a docker image with

project_2

installed as a dependency to project 1 • package + register • run This approach requires rebuilding the docker image with every execution because fast register will not realise that the launchplan defined in

project_2

should have a different version than the rest of the workflow and it fails while trying to fetch the workflow. Another option would be to do something like:

remote = FlyteRemote(config=Config.auto())
launchplan = remote.fetch_launch_plan(...)

but that would require my pod to be able to reach flyteadmin because it's inside a dynamic workflow, Is there anything I can do to avoid rebuilding the docker image for every run? Am I doing something wrong? Any advice is much appreciated. Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn't provide any link to the newly spawned workflows so I can't reliably track the intermediate states or see why something failed (if it did). (see screenshot, the purple task is dynamic). It also leads to all sorts of mixed status reports (see other screenshot).

hey, I’m just confused about hosted version. Do you offer hosted version and how much does it cost?

Hi there, does anyone have any tips on debugging why workflows are stuck in queue indefinitely?

Hi team, We noticed a new UI bug for map tasks, despite all sub-tasks finishing, some “sub-tasks” are still showing as Running in the UI. Is this a known bug?

How to use s3 with flyte spark? Spark with flyte itself works fine with the spark serviceaccount (spark-pi example from the flyte docs) and I tested the spark serviceaccount using a spark task that simply uses boto3. But I can't seem to make something like spark.read.parquet("s3://<bucket>/<path to file>) to work. I tested it with pyspark using the workflow dockerfile and it works if I do aws configure first.

An error occurred while calling o125.parquet.
: java.nio.file.AccessDeniedException: s3://<bucket>/<path>: getFileStatus on s3://<bucket>/<path>: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;

My spark config in values.yaml is:

spark-config-default:
            # We override the default credentials chain provider for Hadoop so that
            # it can use the serviceAccount based IAM role or ec2 metadata based.
            # This is more in line with how AWS works
            - spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
            - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2"
            - spark.kubernetes.allocation.batch.size: "50"
            - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl"
            - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
            - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A"
            - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
            - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A"
            - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
            - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
            - spark.hadoop.fs.s3a.multipart.threshold: "536870912"
            - spark.blacklist.enabled: "true"
            - spark.blacklist.timeout: "5m"
            - spark.task.maxfailures: "8"

I can specify the AWS credentials manually and use "spark.hadoop.fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" in the spark config, but I was wondering if you can use IRSA to avoid using credentials.

#ask-the-community how to run the launch plan in remote ? i m trying with a simple hello world script to send email notification. import typing from flytekit import Email, FixedRate, LaunchPlan, PagerDuty, Slack, WorkflowExecutionPhase, task, workflow @task def say_hello() -> str: return "hello world" @workflow def my_wf() -> str: res = say_hello() return res my_notifying_launch_plan = LaunchPlan.get_or_create( name="my_notifying_lp", workflow=my_wf, notifications=[ Email( phases=[WorkflowExecutionPhase.SUCCEEDED], recipients_email=["kamakshi.muthukrishnan@outlook.com"], ), Slack( phases=[WorkflowExecutionPhase.SUCCEEDED, WorkflowExecutionPhase.FAILED, WorkflowExecutionPhase.ABORTED, WorkflowExecutionPhase.TIMED_OUT,], recipients_email=["XXXX@XXX.slack.com"], ) ], )

Hi 🙂 We are trying to build a Spark Task, and we noticed some behaviors that we don’t quite understand. When running the code from this example : https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html , everything works fine. We would like to leverage Fast registration (https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/deploying_workflows.html#fast-registration) and use it to avoid having to rebuild/repush a Docker image. But then, if we try to change the code in, say, the function

f

, and then use

pyflyte --pkgs path.to.code package --image my_image --force --fast

to package the code and then upload and run the new version of the workflow, we notice that the behavior

f

is the same as before. As I understand it now, this could be because the Spark executors still have the previous code. Does this work as intended ? TL;DR : can we use

--fast

to avoid building / pushing Docker images for Spark tasks ?

What is the recommended way to share curated flows / tasks within an organisation? To add more perspective: In kubeflow we built a bunch of premade components (basically a bit of code + container) that we could load into any pipeline and use in any pipeline. Does something like this exist in Flyte? Best case still executable locally?

#ask-the-community We are dealing with flyte in production generating enormous amounts of data in the S3 flytedata folder. Is there any housekeeping script or anything which the community is using to remove data from the flyte s3 and correspondingly delete entries from the postgres DB's

Has anyone worked with or have any docs on automated workflow registration to a remote flyte cluster, particularly with ArgoCD?

Hi, I have just joined this community after spending about 4 hours watching thru tutorial videos and reading the flyte documents including integrations. In my use case, I have tasks of a DAG that need to be invoked across two k8s clusters. I can do this in Airflow with the use of KubernetesJobOperator, specifying the path to the kuebconfig. This way, task1 is run on cluster1 and once it finishes, task2 is run in cluster2. In Flyte, Is there a way to specify in the task decorator, which k8s cluster (or which kubeconfig file to use) to run on? Thanks!

Hi all, my flyte cluster was up and running on GCP I tried to upgrade it to install bigquery and tensorflow-operator plugin using command

helm upgrade -n flyte flyte -f values-override.yaml ./flyte/charts/flyte-core

. Pods are unable to come up and are crashing since then with below error: Error from

syncresources

pod:

2023/02/10 06:54:59 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/database.go:114
[error] failed to initialize database, got error failed to connect to `host=postgres user=postgres database=flyteadmin`: hostname resolving error (lookup postgres on <ip address>:53: no such host)
Error: failed to connect to `host=postgres user=postgres database=flyteadmin`: hostname resolving error (lookup postgres on <ip address>:53: no such host)

Can anyone help me with this?

How can ensure if a particular plugin is actually installed properly. I followed this doc to install tensorflow-operator but after installation when I tried to launch the example workflow mentioned in the doc, I see below error in my flyteworkflow deployment.

message: >-
    failed at Node[n0]. RuntimeExecutionError: failed during plugin execution,
    caused by: failed to execute handle for plugin [tensorflow]: found no
    current condition. Conditions: []

Hey everyone! I'm working to deploy Flyte in our GKE cluster via the standalone cluster documentation. We have requirements around leveraging Minio (already deployed on the cluster). I am able to get Flyte stood up and access it via console, however, when I "pyflyte run --remote" a workflow I get errors regarding connection to Minio based on the conneciton details I passed into the Flyte deployment via the helm chart. Is the pyflyte cli trying to connect directly to Minio and causing this error? I've tried adjusting the config.yaml with a storage config to point to the port-forwarded minio service but it still appears to try to connect to minio via teh config passed to the deployment.

Original exception: HTTPConnectionPool(host='signals-minio', port=9000): Max retries exceeded with url: /flyte/flytesnacks/development/YK2ZGX4HTNQPDYSEHLOIX7K2UM%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio_access_key%2F20230210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230210T152959Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11d5884c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Hi.. an special configuration is needed to use ECR? flyte is not able to pull the image but when pull the image using docker works fine. I’m getting this: Back-off pulling image

Hello, I have the setup env vars in flyte remote server and can retrieve them via os.environ.get(“env-var”) and also can get the domain value via os.environ.get(“FLYTE_INTERNAL_TASK_DOMAIN”) in task running on the flyte remote. However, this will not work if I run the task locally, because the keys are not set locally. How do I configure my local machine if I want to run the tasks locally to retrieve the same env-vars and FLYTE_INTERNAL_TASK_DOMAIN (pyflyte run locally)?

Hello, In Airflow there is a mechanism for a dag to wait until another dag has successfully completed (ExternalTaskSensor), then it will start. What is the mechanism in Flyte that achieves this?

Good evening! 🙂 What is the mechanism for limiting what projects/domains a registered user should be able to access? I.e. is there any mapping from jwt claims or similar?

Hi all. A while back I mentioned some pain points I feel related to scheduled workflow executions. Some of these can be filed under "things I miss about Airflow", but they are more generally related to backfills and scheduled launch plans. I was asked to write up a list and put it in a feature request. But I thought I'd start here to gather any other ideas before getting to a specific set of feature requests. Here are a few thoughts (not at the level of specificity of a design doc): • Problem: When creating a new scheduled launch plan, I often want to backfill to some given date. ◦ Potential approach: Include an optional argument

start_date

for

CronSchedule

that indicates how far back in the past the time based view should go. All past workflow executions will be scheduled when this is registered, perhaps sequentially. ◦ Defaults to a value based on the time the schedule was activated. That is, by default there is no backfill done. ◦ This determines the number of backfill "slots" to be executed. • Problem: The workflow UI shows the list of workflow executions based on the time they were run. But for scheduled launch plans, I want to see a view based on the scheduled times. How do I know if a particular failed schedule workflow execution ever had a subsequent success? ◦ Potential approach: Have a "schedule time" based view for each schedule. ◦ Visually display the status of each execution slot in the backfill by the status of the last execution. E.g., success, failure, running, etc. ◦ Click on an execution slot to relaunch a failed or successful task or launch a task that has not yet run. • Problem: Running backfills for a scheduled launch plan is tedious. Is there a better way than just a lot of clicking in the UI? ◦ Potential approach: Provide backfill capabilities in the UI for scheduled launch plans. For example, select a launch plan with a schedule, select a start datetime and end datetime, then run. ◦ Provide a few concurrency and backfill settings. (Borrowed from Airflow) ◦

max_active_runs

sets the maximum number of executions running concurrently. Setting to 1 would backfill sequentially, one at a time. • Problem: In the workflow UI, in the section "All Executions in the Workflow", there is no easy way to distinguish between scheduled runs and manually triggered runs. And the inputs to the execution are not readily available in this view. ◦ Potential approach: Enable a filtered view of only those executions that were initiated by a schedule (rather than by the user launching via the UI). ◦ Filter by the name of the launch plan used to trigger the workflow execution. ◦ Include columns for the value of the time based argument passed to the workflow. And or include the name of the launch plan used to trigger the execution. I'd be happy to file these as feature requests, but I'd welcome any feedback before I get started.

Hi all, I have a project where a simulation SIL is run and exported signals from the simulation are processed in python. The simulation happens inside a container. The simulation container is exchanged more frequently than the python scripts are adapted. I read in the Flyte documentation that the task is hard-wired to a container image upon registration. Is it possible to define a container image as input for a task? So that someone can specify the path to the container e.g. within the Flyte GUI? Thank you in advance for your answers

Hmm, when I

flyte-cli sandbox start

should it not server a 30081?

#ask-the-community I am having long running flyteworkflows. I could see that in the FLyteUI i get timeout in NODE. Is there a configuration in FLyte where i can tell flyte to continue the job infinitely till it completes. I am dealing with ML workflows which could run upto 7 days.

Hello everybody, I´ve played around with flyte on my local machine. Now I wanna run some simple (scheduled) workflows on my VPS. I am currently a little confused, on how to do that. Should I simply run a flyte demo cluster (

flyte demo start

) on my VPS?

what the is default project name when you’re running flyte locally?

i get

"failed to validate that project [studio] and domain [development] are registered, err: [project [studio] not found]\"}\"\n>>"}

Hey team, good morning! I’m trying to expose Flyte logs from K8s tasks to Datadog. I saw there was a question about this here and I could not find a work around for this. I was wondering: a) Is there a way to set a fixed name for the container running the task? b) Is there a way to set a volume for the pod running the task? I checked the k8s plugin config but could not find an answer for these. Thanks in advance!

Hello team, is there a guide on how to hookup flyte-mlflow to an existing mlflow tracking server? Our goal is to log model parameters and metrics to an existing self hosted mlflow tracking server.