Hi, I’m trying to call a

@dynamic

task from a

map_task

and am seeing a strange error. Is there a limitation I should be aware of?

[1/1] currentAttempt done. Last Error: UNKNOWN::failed to read data from dataDir [<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>]. Error: path:<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>: not found

The attached code fails on our GCS cluster. Other map_tasks that don’t call a dynamic task work fine

hi there! we created a subclass of TypedSchema that writes to other storages besides GCS/local. i'm getting this error when running a workflow that creates this subclass as an output. this worked up until recently, so wondering if anything changed with TypedSchema recently?

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/home/jovyan/.local/lib/python3.7/site-packages/flytekit/common/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/home/jovyan/.local/lib/python3.7/site-packages/flytekit/core/base_task.py", line 514, in dispatch_execute
        ) from e

Message:

    Failed to convert return value for var o0 for function onemodel.models.subscribers._flyte.subscribers_task with error <class 'AttributeError'>: '_TypedJasperSchema' object has no attribute '_remote_path'

Hi! Are there plans on supporting

BranchNode

and

WorkflowNode

in the local executor for tests? (https://github.com/flyteorg/flytekit-java/blob/3cf3ee4ac27a3e1bad231be2fa9a30b8afa[…]/src/main/java/org/flyte/localengine/ExecutionNodeCompiler.java) It's difficult to write tests for our sub-workflows without support for these in the local executor

quick question about the sandbox. If I'd like to run a workflow where the input to that workflow is a flytefile, where can I place that file in the sandbox environment so it can be found? This is what the UI looks like

Welcome, officially, to Flyte, MethaneSAT… Beyond being naturally excited to see the list of Flyte Users/Contributors grow, I get extra excited to see such big ambitious projects with a goal of making life better build on Flyte. To space and beyond, @Nicholas LoFaso & Team! 🛰️ :flyte:

Hello 👋 is there a way to avoid starting a new scheduled workflow if the previous one is still running?

We're seeing flytepropeller auth failures after installation/upgrade (via helm) with a message like this:

E1220 16:10:36.391892       1 workers.go:102] error syncing 'mandant1-development/f3359d6b5cd941830000': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]

It leaves the system in a weird state because there's no hard failure. flytepropeller ist running after all so our monitoring does not trigger an alarm, but no workflow execution is happening. After a flytepropeller restart, it suddenly starts working again. So after some digging I found out why this happens in our setup:

flyte-secret-auth

is populated with

.Values.secrets.adminOauthClientCredentials.clientSecret

during installation which is set to the placeholder

foobar

in

values.yaml

. We set the secret value dynamically though with a helm hook during installation because we need to fetch the real client-secret from Keycloak. That happens only after flytepropeller is deployed and it seems that flytepropeller does not reload the secret on changes. Since

flyte-secret-auth

is managed by helm, this happens again on every

helm upgrade

. I see mainly two (non exclusive) ways to improve this behavior: • Remove the default

clientSecret

and only create

flyte-secret-auth

via helm if the value is actually set. Only mount

flyte-secret-auth

if external auth is enabled. That would cause flytepropeller to fail to start until

flyte-secret-auth

is created by other means. • Trigger a flytepropeller reload when

flyte-secret-auth

changes. Any thoughts on this? Happy to contribute here but I'd like to discuss the best way forward with you first.

Hey I'm having trouble registering a workflow. It looks like the tasks are registering alright:

flytectl register files --project chariot-sdk-test --domain development --archive out.tar.gz --version v2

 ---------------------------------------------------------------------- --------- ------------------------------------------------------------
| NAME (4)                                                             | STATUS  | ADDITIONAL INFO                                            |
 ---------------------------------------------------------------------- --------- ------------------------------------------------------------
| /tmp/register024275035/0_pm.nb.new.ipynb_1.pb                        | Success | Successfully registered file                               |
 ---------------------------------------------------------------------- --------- ------------------------------------------------------------
| /tmp/register024275035/1_new.ipynb_1.pb                              | Success | Successfully registered file                               |
 ---------------------------------------------------------------------- --------- ------------------------------------------------------------
| /tmp/register024275035/2_workflows.new_workflow.nb_to_python_wf_2.pb | Failed  | Error registering file due to rpc error: code =            |
|                                                                      |         | Internal desc = failed to compile workflow for             |
|                                                                      |         | [resource_type:WORKFLOW project:"chariot-sdk-test"         |
|                                                                      |         | domain:"development"                                       |
|                                                                      |         | name:"workflows.new_workflow.nb_to_python_wf" version:"v2" |
|                                                                      |         | ] with err entry not found                                 |
 ---------------------------------------------------------------------- --------- ------------------------------------------------------------
| /tmp/register024275035/3_workflows.new_workflow.nb_to_python_wf_3.pb | Failed  | Error registering file due to rpc error: code =            |
|                                                                      |         | NotFound desc = entry not found                            |
 ---------------------------------------------------------------------- --------- ------------------------------------------------------------

Any ideas why that might happen?

I have a workflow that is successfully writing outputs but the CRD remains in "Workflow Started", the UI shows "Running" and the pod falls into a "NotReady" state. What triggers the CRD to fall into a error or completed status?

Hi all! 🙂 Trying to start a flyte sandbox locally with the command from the Guide but getting

+-------------------------------------+---------+-----------+
|               SERVICE               | STATUS  | NAMESPACE |
+-------------------------------------+---------+-----------+
| flyte-contour-contour-certgen-xggqx | Pending | flyte     |
+-------------------------------------+---------+-----------+

^ it's just stuck there

I've gotten around the map_task "single input" limitation before by using a data class to consolidate all the inputs together, but it doesn't look like I can use those as inputs to the shell task since you can't (or I think you can't) deference multiple times (i.e.

{{ .<http://inputs.my|inputs.my>__input__data_class.input_1 }}

)

Hi all, how can I tell if a map_task is successfully cached? As far as I can tell this bugfix should now allow for caching, but the UI doesn’t give an indication that it was saved.

Anyone here having issues with running the sandbox environmnet + flytesnacks on a M1 Mac?

Follow up question (sorry for the spam): I got the sandbox to work, but the hello world task from flytesnacks fails.

Hi Community, What a year it has been. I am extremely proud of this community and especially how supportive everyone is & how much have we accomplished together. We have grown to many production deployments and seen such amazing use-cases fulfilled by Flyte, that I could not have ever imagined. I think it is still very early for this community. Even tough we started in 2020, 2021 is really when we created the community. Looking ahead I am extremely excited about the launchpad we have created and whats to come. Earlier this year I started a company called union.ai, with a mission to help orchestrate ML and data processing and make it accessible to everyone across the world using Flyte. The team at union is truly dedicated to ensure that every user of Flyte’s voice is heard. We have done over 20+ integration, 100s of features and 1000s of community members of joined us on this journey. In 2022, we promise to continue working even harder to make Flyte the best platform you will use for you orchestration needs. One way to do this to ensure we focus on the user experience and understand what our users are trying to get from the platform. They use Flyte, when they want resource-aware, reliable, serverless, multi-tenant and extensible platform for orchestrating really complex data and ml jobs. Along the way, as the community scaled, we at union had to choose the things that we could focus on and we had to let some potential new users go (as we could not fulfil their requests). But, with 2022, we will be working on more of these use cases and many more integrations and improvements (stay tuned). Wishing all of you a Merry Christmas and a happy new year! Yours Gratefully, Ketan

hey I am working on a backend webapi go plugin which runs a BigQuery query. I am trying to write an output for the task so whenever the query is successful I use

outputWriter.Put(ctx, ioutils.NewInMemoryOutputReader(outputs, nil))

to store a

outputs

structure to be visible in console. I do this in

Status

method (in this interface) whenever the job is completed. However it seems the protobuf is not written (thus not visible). I followed the same approach for a core plugin and it worked. Am I doing something wrong here in the way I am using the webapi plugin?

Folks, I am working on a new design that would allow users to run low latency workflows on Flyte and run existing simple python tasks and reduce / ammortize the overhead of running a pod. Would folks here be interested in helping me refine the design and the UX model? Goal: Users should be able to run sub second tasks and yet get the same level of visibility, fault tolerance and simplicity for their code. We might lose some isolation properties that running an individual pod offers, but we will continue to offer isolated environments, versioning and dynamic infrastructure.

Hi! Is there an endpoint to list all executions across projects/domains? Like this, but without the project/domain parameters.

Hi everyone, I've got some questions hoping someone can give me a hand. Is there a way to control what gets registered? There are two scenarios in that this is helpful. 1. I have a workflow that has 2 sub-workflows, the sub workflows should not be used separately so I don't want them to be registered and visible in Flyte 2. How can I register a dynamic workflow? Thanks in advance.

Hi All, is there an example for how to configure the python logging package for use with

@tasks

? I’m configuring the root logger, but it seems like that code is only running at registration time not execution time? See thread for example

Hello everyone, Welcome back to our OSS Sync Ups! Join us tomorrow 1/11 at the adjusted time of 7:00pm PT (10:00am WIB, 8:30am IST on 1/12) to learn how Gojek uses Flyte for their ML platform, with @Pradithya Aria Pura. Zoom link: https://us06web.zoom.us/j/85635963908?pwd=N0k0ZUw1MkFadlBvT0VydlkvcXFyZz09 See you then! Flyte Team

Hey everyone! I’m on zoom for the next 30mins hosting Flyte Office Hours. If anyone has questions about deploying/trying/using Flyte, I’ll be happy to discuss!

@Tim Bauer / @Haytham Abuelfutuh can we merge this one- https://github.com/flyteorg/flytesnacks/pull/463

what kubernetes versions (min, max) does flyte support?

Hi team, we are trying to compute the overhead time of the platform. Meaning:

1 - (All workflows transition time / all workflows duration)

We use this PromQL query:

1 - 
  ( flyte:propeller:all:workflow:completion_latency_unlabeled_ms_sum
    + flyte:propeller:all:node:transition_latency_unlabeled_ms_sum
    + flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum
    + flyte:propeller:all:workflow:acceptance_latency_unlabeled_ms_sum)
  / (flyte:propeller:all:workflow:failure_duration_unlabeled_ms_sum + flyte:propeller:all:workflow:success_duration_unlabeled_ms_sum)

The numbers we get seem a bit off ( around 50%). Is the PromQL query correct? What do you recommend we use to measure that overhead time ?

How do I configure default resource limits for tasks?

Hello All. Hope you are doing well. I am a software developer (precisely python developer) and new to the domain of ML. During the journey, i encountered an error while applying flyte for workflow. i am working with spacy and at first it was just one file without any self defined functions and just calling 3rd party APIs. so i somewhat changed one sequential code to function to create tasks to get better understanding of flyte. well, in one function (task) i am returning a list of tuple. like traindata = [...('some text here',{'key': (1,3,'abc')})...] and wrote return type as typing.List[tuple]. but it is giving me an exception that generic types cannot be there. what should i do in this condition

and another question, if i somehow manage to make work perfectly fine, what is the benefit i am getting from that?

Hi team, I am a DS from gojek, currently working on using Flyte for model pipeline. One question I came across is that the resource requested (cpu, memory) is hard coded in the decorator for each task for the current implementation, so it’s the same for different environments. What will be a better way to do overrides for each task in the workflow for different environments? Will it be possible to define it in a config file instead? cc @Pradithya Aria Pura @Michael Cheng Jan Kao