Hello! I am trying to use the Flyte Spark plugin with the SynapseML package with the local runner. It works until I try to add the

spark.jars.packages

to the config (see example below). There is some kind of connection refused error:

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:

. I imagine this has to do with some configuration that I don’t quite understand in the backend. Any help is much appreciated. I tested it out with another package and it had the same error.

import datetime
import random
from operator import add

import flytekit
from flytekit import Resources, task, workflow


from flytekitplugins.spark import Spark

@task(
    task_config=Spark(
        # this configuration is applied to the spark cluster
        spark_conf={
            "spark.driver.memory": "8g",
            "spark.jars.repositories": "<https://mmlspark.azureedge.net/maven>", 
            "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5"  # adding this causes problems
        }
    ),
    limits=Resources(mem="2000M"),
    cache_version="1",
)
def hello_spark(partitions: int) -> float:
    print("Starting Spark with Partitions: {}".format(partitions))

    n = 100000 * partitions
    sess = flytekit.current_context().spark_session
    count = (
        sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
    )
    pi_val = 4.0 * count / n
    print("Pi val is :{}".format(pi_val))
    return pi_val

Hi Flyers, Join us tomorrow for our very first Fireside Chat with Flyte Contributors, plus community updates and roadmap items in our biweekly community sync. Enjoy guest appearances from: • @Sugato Ray, PhD Candidate & UnionML contributor • @Matheus Moreno, ML Engineer at Hurb • @Robin Kahlow, ML Engineer at Ntropy • @Mike Zhong, SR Software Engineer at Embark Veterinary • @krishna Yerramsetty, Data Scientist at Infinome Biosciences Tuesday 7/12 at 9am PT Calendar Invite and Zoom Link See you there! Flyte Team

Hi! We are currently trying to add automatic tracing through datadog to the pods started through Flyte. To do so, we would need to set a few environment variables and mount a volume. While we can set the environment variables in the config, there is no way to add volumes here. Is there any other way that we're missing? The only other thing I could think of is switching all of our tasks to pod tasks, where we would have control over the whole spec, but that is something we would like to avoid if possible. Also happy to contribute here if this is something you'd see a need for.

We've been putting flyte through its paces on high fan-out workflows (ie ~50 parallel subworkflows using

@dynamic

) and we've noticed some flakiness w/ the graph view when expanding subworkflows. In lieu of that, it seems like the timeline view is better suited to finding failing nodes. Would there be interest in a contribution to improve filtering in the timeline view (ie. using the same categories in the graph view)?

Is the name of the workflow (i.e. the function decorated with

@workflow

) the unique identifier for that workflow (combined with project and domain)?

i.e. you can’t have two workflows with the same name?

hello! assuming that we're in a

@dynamic

workflow, is there a way to define dependencies between tasks without relying on their respective outputs? i.e, we would like to run

task2()

after

task1()

even though the latter does not output anything. In a classic

@workflow

, we could rely on

create_node()

+

>>

but can't figure out the equivalent for

@dynamic

Hi Flyers, Flyte will be at SciPy 2022 this week! @Niels Bantilan will talk about the basics of Flyte and its programming model that gives it productionizing strengths, in "*Reliable, Reproducible, Recoverable, and Auditable Machine Learning for Production Workloads with Flyte*" Thursday 7/14, 3:50pm CDT

Developing some memes for Scipy… what do you think?

Would there be any way to set up gang scheduling within a project/domain currently? I've seen something similar for Kubeflow's training operators using https://github.com/volcano-sh/volcano and was wondering if it'd be possible to achieve for Flyte workflows. For context, we'd like to create user-specific projects and enforce resource quotas on them in K8s; however, if we do this, we'd like to try to avoid multiple queued workflows contending for resources and let pods associated w/ a specific workflow get scheduled together. Another example of this would be https://github.com/palantir/k8s-spark-scheduler

Hi Flyers, Thank you to everyone who attended this week's Flyte OS Community Sync. The following recordings have been published: •

Community Update▾

- Martin Stein •

Flyte Signaling and Gate Nodes▾

- Dan Rammer • Fireside Chat: Recording editing and processing in progress. • Meeting notes and slides Keep an eye out for our upcoming Biotech-themed Community Sync on July 26th. See you there! Flyte Team

Is there any established process/scripts [ and associated testing ] for syncing boilerplate ( there is https://github.com/flyteorg/flyte/issues/2561 ). At the moment, I’m specifically interested in https://github.com/flyteorg/boilerplate/tree/master/boilerplate/flyte/end2end being different from what is in https://github.com/flyteorg/flyte/tree/master/boilerplate/flyte/end2end …. Imagining that these are meant to be the same?

Hello Flyte team, While running map tasks, few tasks successfully completed but few were in running state in flyte console for a long time. I checked GCP logs and it showed that the map task completed. kubectl also showed all pods had completed executing except one which was

terminating

. What could be the reason for this? Did the map task not complete because of the one pod ?

Kubectl output

vijay.jaishankervijay@MacBook-Pro ~ % kubectl get pods -n dev | grep ah62ztnhshgx4gqqfgdv
ah62ztnhshgx4gqqfgdv-n0-0                 0/1     Completed                  0          50m
ah62ztnhshgx4gqqfgdv-n1-0                 0/1     Completed                  0          46m
ah62ztnhshgx4gqqfgdv-n2-0-0               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-1               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-10              0/1     Completed                  0          34m
ah62ztnhshgx4gqqfgdv-n2-0-11              1/1     Terminating                0          34m
ah62ztnhshgx4gqqfgdv-n2-0-12              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-13              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-14              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-15              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-16              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-17              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-18              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-19              0/1     Completed                  0          30m
ah62ztnhshgx4gqqfgdv-n2-0-2               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-20              0/1     Completed                  0          28m
ah62ztnhshgx4gqqfgdv-n2-0-21              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-22              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-23              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-24              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-25              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-26              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-27              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-28              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-29              0/1     Completed                  0          23m
ah62ztnhshgx4gqqfgdv-n2-0-3               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-30              0/1     Completed                  0          19m
ah62ztnhshgx4gqqfgdv-n2-0-31              0/1     Completed                  0          19m
ah62ztnhshgx4gqqfgdv-n2-0-32              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-33              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-34              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-35              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-36              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-37              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-38              0/1     Completed                  0          18m
ah62ztnhshgx4gqqfgdv-n2-0-39              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-4               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-40              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-41              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-42              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-43              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-44              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-45              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-46              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-47              0/1     Completed                  0          13m
ah62ztnhshgx4gqqfgdv-n2-0-48              0/1     Completed                  0          6m55s
ah62ztnhshgx4gqqfgdv-n2-0-49              0/1     Completed                  0          6m56s
ah62ztnhshgx4gqqfgdv-n2-0-5               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-6               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-7               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-8               0/1     Completed                  0          42m
ah62ztnhshgx4gqqfgdv-n2-0-9               0/1     Completed                  0          42m

hello! Is there a mean to add custom tags on the Flyte provided metrics mentioned there? The only available tag is currently the workflow name (

wf

), but we would like to add dimensions to the relevant metrics depending of the workflow inputs or the custom labels provided while submitting the workflow. Those labels are only added on the relevant pods as far as I saw but that would be great if those labels could be somehow added to the metrics as well

Hi all! I'm running into an issue where I'm trying to pass in a boolean value to a workflow. I'm on flytekit 1.1.0 and trying to run this command:

pyflyte run my_task.py download --remote # this works
pyflyte run my_task.py download # this does not work (remote=False)

Here's an example of my workflow definition

@workflow
def download(remote: bool = False):
    ...

And here's the error I'm getting

line 191, in to_python_value
    raise TypeTransformerFailedError(f"Cannot convert literal {lv} to {self._type}")
flytekit.core.type_engine.TypeTransformerFailedError: Cannot convert literal scalar {
  primitive {
  }
}
 to <class 'bool'>

Is there anything obvious that I'm doing wrong?

Hey everyone, Join us tomorrow for another Flyte Ecosystem Project UnionML OSS Planning Event. Bring your ideas, questions, and wish list! Wednesday 7/20 at 9am PT Calendar Invite and Zoom Link Also, check out UnionML on GitHub and read the docs. See you then!

Hi Flyers, If you haven't already, check out the Flyte 1.1.0 milestone release featuring: - The all-new FlyteDecks - FlyteKit Optional type support, PyFlyte Run and PyFlyte Register - Backend improvements & bug fixes - UI improvements

Is there a way to skip the y/n confirmation with flytectl?

This action will overwrite an existing config file at [/.flyte/config.yaml]. Do you want to continue? [y/n]:

While running

flytectl config init --host=<http://example.com|example.com>

After setting up a cluster, with metallb + contour ingress (default install with helm flyte sandbox instructions), deploying a sample with command line receives a grpc error:

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
{"asctime": "2022-07-20 21:46:45,700", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.700485000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.700485000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-07-20 21:46:45,906", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.905927000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.905926000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/Users/btyeung/.virtualenvs/flyte/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 516, in _run
    remote_entity = remote.register_script(
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/remote/remote.py", line 596, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 116, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 42, in handler
    return fn(*args, **kwargs)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 859, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1658378806.310949000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":[{"created":"@1658378806.310944000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>

The loadbalancer (metallb) successfully provided an IP address, and console is available through that address. Here is flyte config.yaml

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///192.168.111.220
  insecure: true
logger:
  show-source: true
  level: 0

Here is a screenshot of the console:

Hi! I've been looking into the docs for Notifications (and also checked the corresponding admin code). From what I could gather, GCP support for creating events (in Pub/Sub) already exists, but there is currently no way of sending email notifications with GCP (i.e. no

emailer

for GCP). Just wanted to confirm whether that is correct or if I am missing something?

Hi Flyers, ⚡Flyte will be featured tomorrow 7/22 on Kelsey Hightower's Twitter Space: Machine Learning in Production⚡ Join @Ketan (kumare3) and @Eduardo Apolinario (eapolinario), plus guest speakers @Fabio Grätz, @Evan Sadler, @Arno Hollosi, Jordan Ganoff and Gautam Kedia for insights into unified machine learning and data orchestration. 7/22 at 10am PT Event link (also QR code in the flyer)

👋 Hi everyone!

👋 Hello everyone. I'm exploring flyte as a potential tool for my team and have a question I can't seem find an answer to: Is there any mechanism for configuring the python runtime environment for a task? Other tools I've explored have allowed me to pass in a list of packages (or a requirements.txt) and I'm looking for something similar.