Evan Sadler
07/11/2022, 4:15 PMspark.jars.packages
to the config (see example below). There is some kind of connection refused error: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
.
I imagine this has to do with some configuration that I don’t quite understand in the backend. Any help is much appreciated. I tested it out with another package and it had the same error.
import datetime
import random
from operator import add
import flytekit
from flytekit import Resources, task, workflow
from flytekitplugins.spark import Spark
@task(
task_config=Spark(
# this configuration is applied to the spark cluster
spark_conf={
"spark.driver.memory": "8g",
"spark.jars.repositories": "<https://mmlspark.azureedge.net/maven>",
"spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5" # adding this causes problems
}
),
limits=Resources(mem="2000M"),
cache_version="1",
)
def hello_spark(partitions: int) -> float:
print("Starting Spark with Partitions: {}".format(partitions))
n = 100000 * partitions
sess = flytekit.current_context().spark_session
count = (
sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
)
pi_val = 4.0 * count / n
print("Pi val is :{}".format(pi_val))
return pi_val
Sandra Youssef
07/11/2022, 7:39 PMBernhard Stadlbauer
07/12/2022, 6:21 AMRahul Mehta
07/12/2022, 10:42 PM@dynamic
) and we've noticed some flakiness w/ the graph view when expanding subworkflows. In lieu of that, it seems like the timeline view is better suited to finding failing nodes. Would there be interest in a contribution to improve filtering in the timeline view (ie. using the same categories in the graph view)?seunggs
07/13/2022, 3:03 AM@workflow
) the unique identifier for that workflow (combined with project and domain)?seunggs
07/13/2022, 3:04 AMPrada Souvanlasy
07/13/2022, 11:32 AM@dynamic
workflow, is there a way to define dependencies between tasks without relying on their respective outputs? i.e, we would like to run task2()
after task1()
even though the latter does not output anything.
In a classic @workflow
, we could rely on create_node()
+ >>
but can't figure out the equivalent for @dynamic
Rémy Dubois
07/13/2022, 3:59 PMSandra Youssef
07/13/2022, 5:45 PMNiels Bantilan
07/14/2022, 3:48 PMRahul Mehta
07/15/2022, 6:03 PMSandra Youssef
07/15/2022, 8:26 PMCommunity Update▾
Flyte Signaling and Gate Nodes▾
austin
07/15/2022, 8:50 PMVijay Saravana
07/16/2022, 1:26 AMterminating
. What could be the reason for this? Did the map task not complete because of the one pod ?Vijay Saravana
07/16/2022, 1:28 AMVijay Saravana
07/16/2022, 1:30 AMvijay.jaishankervijay@MacBook-Pro ~ % kubectl get pods -n dev | grep ah62ztnhshgx4gqqfgdv
ah62ztnhshgx4gqqfgdv-n0-0 0/1 Completed 0 50m
ah62ztnhshgx4gqqfgdv-n1-0 0/1 Completed 0 46m
ah62ztnhshgx4gqqfgdv-n2-0-0 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-1 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-10 0/1 Completed 0 34m
ah62ztnhshgx4gqqfgdv-n2-0-11 1/1 Terminating 0 34m
ah62ztnhshgx4gqqfgdv-n2-0-12 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-13 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-14 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-15 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-16 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-17 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-18 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-19 0/1 Completed 0 30m
ah62ztnhshgx4gqqfgdv-n2-0-2 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-20 0/1 Completed 0 28m
ah62ztnhshgx4gqqfgdv-n2-0-21 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-22 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-23 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-24 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-25 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-26 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-27 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-28 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-29 0/1 Completed 0 23m
ah62ztnhshgx4gqqfgdv-n2-0-3 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-30 0/1 Completed 0 19m
ah62ztnhshgx4gqqfgdv-n2-0-31 0/1 Completed 0 19m
ah62ztnhshgx4gqqfgdv-n2-0-32 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-33 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-34 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-35 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-36 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-37 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-38 0/1 Completed 0 18m
ah62ztnhshgx4gqqfgdv-n2-0-39 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-4 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-40 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-41 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-42 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-43 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-44 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-45 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-46 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-47 0/1 Completed 0 13m
ah62ztnhshgx4gqqfgdv-n2-0-48 0/1 Completed 0 6m55s
ah62ztnhshgx4gqqfgdv-n2-0-49 0/1 Completed 0 6m56s
ah62ztnhshgx4gqqfgdv-n2-0-5 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-6 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-7 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-8 0/1 Completed 0 42m
ah62ztnhshgx4gqqfgdv-n2-0-9 0/1 Completed 0 42m
Prada Souvanlasy
07/18/2022, 8:59 AMwf
), but we would like to add dimensions to the relevant metrics depending of the workflow inputs or the custom labels provided while submitting the workflow. Those labels are only added on the relevant pods as far as I saw but that would be great if those labels could be somehow added to the metrics as wellEric Hsiao
07/18/2022, 5:31 PMpyflyte run my_task.py download --remote # this works
pyflyte run my_task.py download # this does not work (remote=False)
Here's an example of my workflow definition
@workflow
def download(remote: bool = False):
...
And here's the error I'm getting
line 191, in to_python_value
raise TypeTransformerFailedError(f"Cannot convert literal {lv} to {self._type}")
flytekit.core.type_engine.TypeTransformerFailedError: Cannot convert literal scalar {
primitive {
}
}
to <class 'bool'>
Is there anything obvious that I'm doing wrong?Sandra Youssef
07/19/2022, 7:19 PMSandra Youssef
07/20/2022, 7:33 PMseunggs
07/21/2022, 3:58 AMThis action will overwrite an existing config file at [/.flyte/config.yaml]. Do you want to continue? [y/n]:
seunggs
07/21/2022, 4:00 AMflytectl config init --host=<http://example.com|example.com>
Bryan Yeung
07/21/2022, 5:40 AMpyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
{"asctime": "2022-07-20 21:46:45,700", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.700485000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.700485000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-07-20 21:46:45,906", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.905927000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.905926000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
File "/Users/btyeung/.virtualenvs/flyte/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 516, in _run
remote_entity = remote.register_script(
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/remote/remote.py", line 596, in register_script
upload_location, md5_bytes = fast_register_single_script(
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 116, in fast_register_single_script
upload_location = create_upload_location_fn(content_md5=md5)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 42, in handler
return fn(*args, **kwargs)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 859, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1658378806.310949000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":[{"created":"@1658378806.310944000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>
Bryan Yeung
07/21/2022, 5:41 AMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///192.168.111.220
insecure: true
logger:
show-source: true
level: 0
Here is a screenshot of the console:Bryan Yeung
07/21/2022, 5:47 AMBernhard Stadlbauer
07/21/2022, 2:24 PMemailer
for GCP). Just wanted to confirm whether that is correct or if I am missing something?Sandra Youssef
07/21/2022, 7:27 PMJordan Ganoff
07/21/2022, 9:39 PMJosh Kilts
07/21/2022, 10:02 PMGeorge Snelling
07/22/2022, 12:06 AMGeorge Snelling
07/22/2022, 12:06 AM