Sam Peers Nitzberg
04/27/2022, 7:18 PMSam Peers Nitzberg
04/27/2022, 7:19 PMAlex Bain
04/28/2022, 12:48 AMv1.0.0
that have partially broken the kubernetes_service_account
, i.e. some of our workflows are executing with the declared service account and some are just executing with default
. I'm still working on debugging it.Ahmad Islam
04/28/2022, 6:50 AMTim Bauer
04/28/2022, 2:42 PMHafsa Junaid
04/28/2022, 8:53 PMAhmad Islam
04/29/2022, 5:10 AMMultiple Image Based Implementation in Flyte
. I can't see the implementation in docs as it is left as todo and I'm new to Flyte
.Sandra Youssef
Ahmad Islam
04/29/2022, 7:12 PMmap volumes
through flyte from a directory inside VM to a directory inside Container and vice versa.(Bind mount)Ahmad Islam
05/01/2022, 6:47 PMRaw containers cannot be run locally at the moment.
What does this statement mean? So, even if I have local images, I am unable to implement Flyte on them?Sandra Youssef
Alex Pozimenko
05/02/2022, 6:17 PMBernhard Stadlbauer
05/03/2022, 7:37 AM@dataclass_json
@dataclass
class Foo:
bar: int
baz: float
@workflow
def my_wf(foo: Foo):
...
Is it possible to introspect the fields of the STRUCT
or is this checked at runtime?Babis Kiosidis
05/03/2022, 8:02 AMSören Brunk
05/03/2022, 9:08 AMflyteadmin:1.0.1
for the stow fix https://github.com/flyteorg/flyteadmin/pull/416 but now flyteconsole fails with a 500.
"Request failed with status code 500 header key "userinfo" contains value with non-printable ASCII characters"Hampus Rosvall
05/03/2022, 1:33 PMSandra Youssef
Matheus Moreno
05/04/2022, 9:06 PM@task_setup
that, as the name suggests, executes some startup/teardown code before and after each task. This decorator is defined inside a submodule tasks.utils.flyte
. My tasks are inside the tasks
module, separated by different modules. When I packaged the tasks before, their names were `tasks.<module>.<task_name>`; now, their names are tasks.utils.flyte.<task_name>
. Basically, the path for the decorator is being used as the "prefix" for the task. This is not ideal and the old method makes way more sense... What should I do?Matheus Moreno
05/04/2022, 10:56 PMflyte-sandbox
(I just altered the ports):
services:
sandbox:
image: <http://cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565|cr.flyte.org/flyteorg/flyte-sandbox-lite:sha-81044f8b1d620b123318db3fb9fe8d38fe74f565>
container_name: flyte-sandbox
privileged: true
ports:
- "30080:30080" # Console at <http://localhost:30080/console>
- "30081:30081"
- "30082:30082"
- "30084:30084"
- "30086:30086"
- "30088:30088"
- "30089:30089"
austin
05/05/2022, 2:52 AMWilliam Burke
05/05/2022, 2:53 PMError: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.securityContext): unknown field "fsGroupChangePolicy" in io.k8s.api.core.v1.PodSecurityContext
Sandra Youssef
Eduardo Apolinario (eapolinario)
05/06/2022, 7:47 PMMatheus Moreno
05/06/2022, 8:01 PMcontainers with unready status: [ash8dlh4m294mnd9psbs-f62iya2y-0]|Back-off pulling image "flyte-example:1.0"
when trying to execute a workflow, even though I have the flyte-example:1.0 image on my local repo. But, at the same time, it doesn't seem like I should have to build the image inside the sandbox container, right? The container from flytectl demo start
doesn't even mount the working directory into /root
.
I can't find anything in the documentation that says how to do this...Leonardo Almeida
05/06/2022, 8:10 PMimport typing
import pandas as pd
import numpy as np
from flytekit import task, workflow
@task
def generate_normal_df(n:int, mean: float, sigma: float) -> pd.DataFrame:
return pd.DataFrame({"numbers": np.random.normal(mean, sigma,size=n)})
@task
def compute_stats(df: pd.DataFrame) -> typing.Tuple[float, float]:
return float(df["numbers"].mean()), float(df["numbers"].std())
@workflow
def wf(n: int = 200, mean: float = 0.0, sigma: float = 1.0) -> typing.Tuple[float, float]:
return compute_stats(df=generate_normal_df(n=n, mean=mean, sigma=sigma))
When I run the code using the local cluster using flytectl demo start (k3s) it works. But in production environment using EKS and ALB does not work.
pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2 SIGINT(2) ↵ 10055 17:06:22
{"asctime": "2022-05-06 17:06:31,287", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867591.287595583\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867591.287594125\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-05-06 17:06:32,274", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651867592.274550785\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651867592.274550285\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
File "/usr/local/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/lalmeida/.local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/clis/sdk_in_container/run.py", line 471, in _run
wf = remote.register_script(
File "/usr/local/lib/python3.8/dist-packages/flytekit/remote/remote.py", line 536, in register_script
upload_location, md5_bytes = fast_register_single_script(
File "/usr/local/lib/python3.8/dist-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
upload_location = create_upload_location_fn(content_md5=md5)
File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
return super(SynchronousFlyteClient, self).create_upload_location(
File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 40, in handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/clients/raw.py", line 834, in create_upload_location
return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1651867592.675517306","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1651867592.675516568","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
Do you have any ideas? I'm using helm with values-eks.yamlHampus Rosvall
05/09/2022, 11:59 AMSören Brunk
05/09/2022, 12:35 PMbucket-name/fast/
before (not sure if I had configured it before, I can't find any config related to that). Now it's just stored in random paths under the bucket root.Alex Pozimenko
05/09/2022, 9:02 PMError in plugin [inputs.prometheus]: error reading body: net/http: request canceled (Client.Timeout exceeded while reading body)
original settings were:
interval = "10s"
response_timeout = "3s"
I changed these to 30s and 15s and it seems to be ok for now.
Do you have a recommendation on that these should be set to?