I've deployed flyte using kubeadm along with spark-k8s-operator which run successful this example, did try run the example from flytesnacks and got some issues (logs in 🧵). Someone know about

Non-spark-on-k8s command provided

?

Hello, I’m trying to run the Getting Started wine classification example on my Flyte setup, but have faced several errors. My setup is consisted of: Jupyter Hub + Flyte on the same K8S cluster. The codes are

from flytekit.remote import FlyteRemote
from flytekit.configuration import Config, PlatformConfig, ImageConfig, SerializationSettings

remote = FlyteRemote(
    config=Config(
        platform=PlatformConfig(
            endpoint="dns:///flyte-backend-flyte-binary-grpc.namespace.svc.cluster.local:8089",
            insecure=True,
            insecure_skip_verify=True,
        )
    ),
    default_project="flytesnacks",
    default_domain="development",
    data_upload_location="<s3a://flyte-storage/>"
)

HYPERPARAMS = {
    "hyperparameters": {"C": 0.1}
}

flyte_entity = training_workflow

flyte_workflow = remote.register_workflow(
    entity=flyte_entity,
    serialization_settings=SerializationSettings(image_config=ImageConfig.auto_default_image()),
    version="v1"
)

launch_plan = LaunchPlan.get_or_create(
    workflow=flyte_workflow,
    name="wine_classification_lp",
    default_inputs=HYPERPARAMS,
)
remote.register_launch_plan(launch_plan, "v1", project="flytesnacks", domain="development")

And the workflow failed right at the first task

get_data

, here is the error I’ve got

[1/1] currentAttempt done. Last Error: USER::rapped(*args, **kwargs)                        │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py:347 in     │
│ _execute_task                                                                │
│                                                                              │
│ ❱ 347 │   │   _task_def = resolver_obj.load_task(loader_args=resolver_args)  │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/core/utils.py:295 in wrapper │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /usr/local/lib/python3.8/site-packages/flytekit/core/python_auto_container.p │
│ y:235 in load_task                                                           │
│                                                                              │
│ ❱ 235 │   │   task_module = importlib.import_module(name=task_module)  # typ │
│                                                                              │
│ /usr/local/lib/python3.8/importlib/__init__.py:127 in import_module          │
│                                                                              │
│ ❱ 127 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│ in _gcd_import:1011                                                          │
│ in _sanity_check:950                                                         │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: Empty module name

Any suggestion on which might have caused this error and how to fix it? I’m suspecting that I haven’t registered the workflow or the launch plan properly.

I am currently trying to combine dynamic overrides with ray to dynamically determine the ray cluster size when running a task. But when the task under the dynamic runs, it is considered a python task and never creates a ray cluster.

@dynamic
def request_train_resources_task(
    compressed_dataset: FlyteFile,
    train_parameters: TrainParameters,
) -> TrainOutputs:
    task_config = RayJobConfig(
        head_node_config=HeadNodeConfig(
            ray_start_params={
                "block": "true",
            }
        ),
        worker_node_config=[
            WorkerNodeConfig(
                group_name="ray-group",
                replicas=train_parameters.num_nodes - 1,
            )
        ],
    )
    resources_request = Resources(
        cpu=str(train_parameters.cpus_per_gpu * train_parameters.num_gpus),
        mem=f"{train_parameters.memory_per_gpu * train_parameters.num_gpus}G",
        gpu=str(train_parameters.num_gpus),
        ephemeral_storage=f"{train_parameters.ephemeral_storage}G",
    )
    print(f"task_config: {task_config}")
    print(f"resources_request: {resources_request}")
    return pl_mnist_train_task(
        compressed_dataset=compressed_dataset,
        train_parameters=train_parameters,
    ).with_overrides(
        task_config=task_config,
        requests=resources_request,
    )

Hey folks.. so I'm exploring how to set up the CI/CD pipeline now that I have some workflows running locally. In order to do this I was experimenting with the flyte-sandbox docker image to register the new version of the workflows and so on. I'm running into an error though

flytectl get workflows --admin.endpoint <remote_host>:30080 -p <project_name> -d development

is successful

docker run --rm --entrypoint flytectl <http://cr.flyte.org/flyteorg/flyte-sandbox|cr.flyte.org/flyteorg/flyte-sandbox> get workflows --admin.endpoint <remote_host>:30080 -p <project_name> -d development

throws an error

time="2022-06-27T20:18:25Z" level=info msg="[0] Couldn't find a config file []. Relying on env vars and pflags."
{"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","ts":"2022-06-27T20:18:26Z"}
{"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-06-27T20:18:26Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-06-27T20:18:26Z"}
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
{"json":{},"level":"error","msg":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"","ts":"2022-06-27T20:18:27Z"}

I have a dynamic task, that has a few chained tasks (using >> operator), when running I get the following error`RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path/to/futures.pb: [LIMIT_EXCEEDED] limit exceeded. 22mb > 10mb.` Any thoughts on what might be going on here? Thanks!

I'm trying to understand the difference between task and workflow at a fundamental level. My understanding was that tasks could not call other tasks. I wrote the following dummy example:

@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)

When run locally, we see type-errors because the

echo_integer

task now returns a promise instead of an int. Interestingly, it works fine when we run remotely, with the caveat that the called task is not run as a isolated container. I would have expected it to fail remotely as well. Any pointers about what I'm missing in my understanding?

Anyone using latest

mypy

and have this issue with

NamedTuple

(for named outputs). The example in the docs (shown below) fails mypy 0.910 Code from Flyte docs:

import typing

from flytekit import task

hello_output = typing.NamedTuple("OP", greet=str)


@task
def say_hello() -> hello_output:
    return hello_output("hello world")

Error:

mint/workflows/tmp.py:5:16: error: Unexpected arguments to namedtuple()  [misc]
    hello_output = typing.NamedTuple("OP", greet=str)
                   ^
mint/workflows/tmp.py: note: In function "say_hello":
mint/workflows/tmp.py:10:12: error: Too many arguments for "hello_output"  [call-arg]
        return hello_output("hello world")
               ^

Hey guys, sorry if this is a dumb question but recently been deploying a series of clusters with Flyte but one of them is constantly running into this error:

2023/06/21 16:43:05 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[126.528ms] [rows:-] SELECT c.column_name, constraint_name, constraint_type FROM information_schema.table_constraints tc JOIN information_schema.constraint_column_usage AS ccu USING (constraint_schema, constraint_name) JOIN information_schema.columns AS c ON c.table_schema = tc.constraint_schema AND tc.table_name = c.table_name AND ccu.column_name = c.column_name WHERE constraint_type IN ('PRIMARY KEY', 'UNIQUE') AND c.table_catalog = 'app' AND c.table_schema = CURRENT_SCHEMA() AND c.table_name = 'datasets'
{"json":{"src":"start.go:174"},"level":"panic","msg":"Failed to start Admin, err: database migration failed: ERROR: relation \"description_entities\" does not exist (SQLSTATE 42P01)","ts":"2023-06-21T16:43:05Z"}
panic: (*logrus.Entry) 0xc000492230

goroutine 68 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).log(0xc0004921c0>, 0x0, {0xc001218200, 0x7d})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x45b
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).Log(0xc0004921c0>, 0x0, {0xc001487e68?, 0x1?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).Logf(0xc0004921c0>, 0x0, {0x3071945?, 0x0?}, {0xc0009c3c50?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:338 +0x85
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3f0ca00|github.com/sirupsen/logrus.(*Entry).Panicf(0x3f0ca00>?, {0x3071945?, 0x416667?}, {0xc0009c3c50?, 0x29b5840?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:376 +0x34
<http://github.com/flyteorg/flytestdlib/logger.Panicf({0x3f0ca00|github.com/flyteorg/flytestdlib/logger.Panicf({0x3f0ca00>?, 0xc0009ead40?}, {0x3071945, 0x1e}, {0xc0009c3c50, 0x1, 0x1})
	/go/pkg/mod/github.com/flyteorg/flytestdlib@v1.0.17/logger/logger.go:188 +0x64
<http://github.com/flyteorg/flyte/cmd/single.glob..func4.1()|github.com/flyteorg/flyte/cmd/single.glob..func4.1()>
	/flyteorg/build/cmd/single/start.go:174 +0xcc
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5

Two other clusters were deployed just yesterday and never ran into this error. Hopefully it's something in terms of configuration that I've overlooked but was wondering if anybody had some insight. EDIT: Two other clusters were using Postgres 14.6 and the one erroring out is using 14.7; could that be the source of the error?

Hi, wondering what people are using for viewing/archiving container logs on self managed on prem k8s... did anyone find a nice setup yet, since flyte can't collect and archive task logs for you?

Hey everyone I'm busy working trying to deploy Flyte to k8s hosted on Hetzner but struggling as existing deployment patterns seem to be geared toward AWS / GCP. Flyte looks awesome but at this point I can't even deploy it well enough to test it in our environment. I've tried using the

flyte-binary

helm chart. Also having trouble working around this: https://github.com/flyteorg/flyte/issues/3769 Could use a hand if anyone has one to spare.