hi team while listing signals using flyte remote we are gett Flyte #flyte-support

hi team, while listing signals using flyte remote ...

square-carpet-13590

01/13/2025, 1:41 PM

hi team, while listing signals using flyte remote we are getting below issue

FlyteSystemException: SYSTEM:Unknown: error=None, cause=<_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNIMPLEMENTED

details = "unknown service flyteidl.service.AuthMetadataService"

debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"unknown service flyteidl.service.AuthMetadataService", grpc_status:12,

created_time:"2025-01-13T19:08:28.302826+05:30"}"

with flyteadmin logging wrong column name for signal table

│ 2025/01/13 13:38:28 */go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/signal_repo.go:71 ERROR: column "project" does not exist (SQLSTATE 42703)*                           │

│ [0.535ms] *[rows:0]* SELECT * FROM "signals" WHERE project = 'flyte-pai' AND domain = 'development' AND name = 'aqd6bxqgsh6jhf5kvl7b' LIMIT 100

has anyone faced this ? we are using flyte v1.14.1

square-carpet-13590

01/13/2025, 1:42 PM

using below script

Untitled

square-carpet-13590

01/13/2025, 3:46 PM

its working with flyte version 1.13.3

average-finland-92144

01/13/2025, 5:05 PM

@square-carpet-13590 I'm curious, how do you upgrade flyte? I'm trying to see if we could have logs from the upgrade process and see if something went off during the DB migrations

square-carpet-13590

01/15/2025, 12:18 PM

@average-finland-92144 we are using terraform, also the helm release status is deployed. no sign of any failure

Copy code

resource "helm_release" "flyte" {
  name             = "flyte-core"
  repository       = "<https://flyteorg.github.io/flyte>"
  chart            = "flyte-core"
  namespace        = "flyte"
  version          =  local.TF_ENVIRONMENT == "develop" ? "1.14.1" : "1.14.1"
  create_namespace = true
  values           = [file("${path.module}/flyte/helm/${terraform.workspace}-core.yaml")]
...........
...........

Copy code

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
flyte-core      flyte           19              2025-01-15 17:34:07.044201 +0530 IST    deployed        flyte-core-v1.14.1

square-carpet-13590

01/15/2025, 12:20 PM

along with that flyteadmin container run-migrations completed with success i.e. no failure logs, but the last migrations in flyteadmin db is same as of

"2024-08-08-remove-input-uri-for-start-nodes"

which was also present in v 1.13.2

square-carpet-13590

01/15/2025, 12:21 PM

even i tried running migrate command manually from flyteadmin pod shell as below, but nothing

Copy code

~ $ flyteadmin --config /etc/flyte/config/db.yaml migrate run
INFO[0000] Using config file: [/etc/flyte/config/db.yaml] 
~ $

square-carpet-13590

01/15/2025, 12:28 PM

one more thing @average-finland-92144 after flyte-core deployment we are not seeing ay debug logs, from flyteadmin/flytepropeller, even the config is populated in container at etc/flyte/config/logger.yaml via values as below, vs in binary deployment we are getting debug logs

Copy code

/etc/flyte/config $ cat logger.yaml
formatter:
 type: json
level: 6
mute: false
show-source: true

average-finland-92144

01/15/2025, 2:26 PM

regarding the error with the

signals

table I just upgraded a flyte-binary setup, then moved to flyte-core 1.14.1 and the

signals

table is there

high-accountant-32689

01/15/2025, 3:11 PM

@square-carpet-13590, you found a legit bug. Fix incoming.

square-carpet-13590

01/15/2025, 3:14 PM

@high-accountant-32689 the bug is related to migrations or logs ?

high-accountant-32689

01/15/2025, 3:15 PM

oh, I mean the

Signals

migration issue.

square-carpet-13590

01/15/2025, 3:16 PM

okay Thank you.

square-carpet-13590

01/15/2025, 3:16 PM

cc @square-tomato-39478 @glamorous-rainbow-77959

rough-apple-2545

02/08/2025, 12:45 AM

Hi team, I use the sandbox for a local minimal flyte cluster, and occur the same issue:`SYSTEMUnknown error=None, cause=<_InactiveRpcError of RPC that terminated with:`

status = StatusCode.UNIMPLEMENTED

details = "unknown service flyteidl.service.AuthMetadataService"

debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"unknown service flyteidl.service.AuthMetadataService", grpc_status:12, created_time:"2025-02-08T08:44:48.386769+08:00"}"

The

flytectl sandbox start

command use the docker image cr.flyte.org/flyteorg/flyte-sandbox:dind-8a5f54c06b047db7832292f1a8a9f4a218e97b5c Is there a way to solve this problem? Thank you.

average-finland-92144

02/10/2025, 6:37 PM

@rough-apple-2545 and you're also trying to get

Signals

from FlyteRemote?

high-accountant-32689

02/10/2025, 7:00 PM

can you use

flytectl demo start

instead?

rough-apple-2545

02/11/2025, 3:26 AM

@average-finland-92144 Yes, I try to get the list_signals of FlyteRemote and occur the error:

unknown service flyteidl.service.AuthMetadataService

rough-apple-2545

02/11/2025, 3:28 AM

@high-accountant-32689 While using the

flytectl demo start

, i get the same issue. The docker image is : cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8a5f54c06b047db7832292f1a8a9f4a218e97b5c

high-accountant-32689

02/11/2025, 4:18 PM

oh, sorry, you're correct. This signals migration issue was fixed in https://github.com/flyteorg/flyte/pull/6172, but this hasn't been released yet. We're planning a release for this next Friday.

square-tomato-39478

02/11/2025, 4:20 PM

@square-carpet-13590 let try to update flyte core to latest version next week

6 Views

Open in Slack

Previous Next