I m having trouble accessing flyteadmin from flytectl that i Flyte #flyte-support

I’m having trouble accessing flyteadmin from flyte...

gifted-raincoat-59712

09/14/2022, 11:48 PM

I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in

flyte

namespace • flytectl running in

foo

namespace If I run flytectl from within the pod in

foo

namespace, and hit the external endpoint (through the ingress), it works

Copy code

bin/flytectl get projects \
  --admin.endpoint dns:///EXTERNAL_FQDN:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

However, if I try to hit the internal grpc endpoint, I get nada. No output.

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 

echo $?
0

🤔

tall-lock-23197

09/15/2022, 9:51 AM

@icy-agent-73298, could you help?

icy-agent-73298

09/15/2022, 9:59 AM

Hi @gifted-raincoat-59712 thats strange, can you also add more logging to the command by passing

--logger.level 6

gifted-raincoat-59712

09/15/2022, 11:21 AM

i should have added the logger first thing. seems unauthenticted? odd because it’s the same client secret

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}

icy-agent-73298

09/15/2022, 11:24 AM

These log mention that it attempted and it did retrieve the projects and it contains 1 project.

Copy code

{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}

But it doesn’t print the o/p though . hmm.

icy-agent-73298

09/15/2022, 11:24 AM

Can you try

-o yaml

output format

icy-agent-73298

09/15/2022, 11:29 AM

The client tries unauthenticated communication first https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L63 and then uses creds flow if it fails https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L69 So you can ignore the error log . But the communication did went through in your case and retrieved the data . It strangely failed in last step of printing the data

gifted-raincoat-59712

09/15/2022, 11:35 AM

yaml output is also empty

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.

icy-agent-73298

09/15/2022, 11:38 AM

hmm strange. Do you mind adding more logging on your end to see what is it returning from admin Also mind checking what happens when you hit admin logs https://github.com/flyteorg/flytectl/blob/master/cmd/get/project.go#L102-L103

gifted-raincoat-59712

09/15/2022, 11:49 AM

I’m not really in a position to edit the source to add more logging, if that’s what you’re asking.

icy-agent-73298

09/15/2022, 11:56 AM

yeah i was hoping we could do that. May be we can try other ways . What happens when you pass in project name

dpp

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>.  --logger.level 6 \
>.  dpp

icy-agent-73298

09/15/2022, 11:57 AM

Another you could try is portforwarding the admin and using that instead of

flyteadmin.flyte.svc.cluster.local:81

gifted-raincoat-59712

09/15/2022, 11:59 AM

same empty output

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:58:54Z"}

icy-agent-73298

09/15/2022, 12:00 PM

What if you pass in non-existent project

gifted-raincoat-59712

09/15/2022, 12:03 PM

same

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:02:55Z"}

icy-agent-73298

09/15/2022, 12:04 PM

It expects the arg[0] to be passed as the name of the project. --project flag is ignored in this case

Copy code

--project foo

Can you pass the project the way i sent you

gifted-raincoat-59712

09/15/2022, 12:08 PM

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:07:50Z"}
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp

gifted-raincoat-59712

09/15/2022, 12:11 PM

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp

gifted-raincoat-59712

09/15/2022, 12:11 PM

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

gifted-raincoat-59712

09/15/2022, 12:11 PM

seems a bit odd. passing any argument ends up printing the output

icy-agent-73298

09/15/2022, 12:18 PM

Strange i dont see this behavior .

Copy code

(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects flyteexamples --logger.level=0
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester  | flytetester  | flytetester description  |
 --------------- --------------- --------------------------- 
| flytesnacks  | flytesnacks  | flytesnacks description  |
 --------------- --------------- ---------------------------

Can you move all flags to you config.yaml file . You can define one using

flytectl config init --host=flyteadmin.flyte.svc.cluster.local:81

and update the authType, clientId, clientSecretLocation in that file. Also assuming you have latest flytectl

gifted-raincoat-59712

09/15/2022, 12:26 PM

does the client use different code paths depending on whether there’s a config file?

gifted-raincoat-59712

09/15/2022, 12:26 PM

can you try without a config file?

icy-agent-73298

09/15/2022, 12:30 PM

Same results for me without the config file . There is not much deviation other than reading the flags from the file

Copy code

(lab_project) ➜  flytectl git:(device-auth) ✗ mv ~/.flyte/config.yaml ~/.flyte/config.yaml_bkp
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081              
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081 flyteexamples
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows

icy-agent-73298

09/15/2022, 12:33 PM

have my config.yaml here

~/.flyte/config.yaml

which i have moved to backup

gifted-raincoat-59712

09/15/2022, 12:36 PM

to recap where we are external endpoint, without positional argument = OK

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

external endpoint, with positional argument = OK

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

internal endpoint, without positional argument = FAIL (no output)

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.

internal endpoint, with positional argument = OK

Copy code

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

gifted-raincoat-59712

09/15/2022, 12:37 PM

so how does code path differ for internal endpoint without positional argument?

icy-agent-73298

09/15/2022, 12:37 PM

This one is not working in my opinion correctly since anything you pass is returning you the project

Copy code

internal endpoint, with positional argument = OK

gifted-raincoat-59712

09/15/2022, 12:38 PM

i would agree with that

icy-agent-73298

09/15/2022, 12:40 PM

https://github.com/flyteorg/flytectl/blob/master/cmd/get/project.go#L87-L100

icy-agent-73298

09/15/2022, 12:41 PM

Seems in the positional case we dont call

ProjectToProtoMessages

icy-agent-73298

09/15/2022, 12:46 PM

That function looks harmless to me as it just create a new proto.Message slice of the same length of projects slice and adds the projects proto

Copy code

func ProjectToProtoMessages(l []*admin.Project) []proto.Message {
	messages := make([]proto.Message, 0, len(l))
	for _, m := range l {
		messages = append(messages, m)
	}
	return messages
}

gifted-raincoat-59712

09/15/2022, 12:48 PM

right, but if that block is not called (no arguments), then we’re not seeing any output, which is the original problem

icy-agent-73298

09/15/2022, 12:50 PM

yes

icy-agent-73298

09/15/2022, 12:51 PM

Strange it works well from EXTERNAL endpoint though

gifted-raincoat-59712

09/15/2022, 12:53 PM

btw, this is my version

Copy code

{
  "App": "flytectl",
  "Build": "f18901c",
  "Version": "0.6.13",
  "BuildTime": "2022-09-15 12:53:00.692399756 +0000 UTC m=+0.025439443"
}

gifted-raincoat-59712

09/15/2022, 12:53 PM

and flyte helm chart 1.1.0

icy-agent-73298

09/15/2022, 12:57 PM

those versions look good to me .

icy-agent-73298

09/15/2022, 1:13 PM

adding @acceptable-policeman-57188 if we have seen such cases before. looks a strange issue

icy-agent-73298

09/15/2022, 1:17 PM

@gifted-raincoat-59712 another thing that could help is any other flyte get also encounters similar issue. eg :

flytectl get workflows

gifted-raincoat-59712

09/15/2022, 1:39 PM

here is what i’m really trying to do.

flytectl register files

if i use the external endpoint, it SUCCEEDS

Copy code

bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
...
85 rows

but if i use internal endpoint, FAIL. here it complains about the tarball not being a tar gzipped file, but it is.

Copy code

bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}

file msat-platform-flyte-package.tgz
msat-platform-flyte-package.tgz: gzip compressed data, was "msat-platform-flyte-7b89d7d.tgz", last modified: Thu Sep 15 13:20:16 2022, max compression, original size modulo 2^32 327680

gifted-raincoat-59712

09/15/2022, 1:43 PM

with more log output

Copy code

bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}

icy-agent-73298

09/15/2022, 1:47 PM

Can we keep the registration issue on a separate thread . Could be different issue here

gifted-raincoat-59712

09/15/2022, 1:47 PM

well, that failure led me to run the simplest command i could.

get projects

gifted-raincoat-59712

09/15/2022, 1:48 PM

can i just bypass flytectl? run a curl command or something?

gifted-raincoat-59712

09/15/2022, 1:48 PM

ah. grpc

icy-agent-73298

09/15/2022, 1:49 PM

yeah . i haven’t tried but postman now supports grpc https://blog.postman.com/postman-now-supports-grpc/

gifted-raincoat-59712

09/15/2022, 1:51 PM

i mean from a pod CLI.

icy-agent-73298

09/15/2022, 1:53 PM

Soemthing like this . https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md https://blog.mattchung.me/2020/11/11/testing-your-grpc-services-using-grpc_cli/

gifted-raincoat-59712

09/15/2022, 1:54 PM

then again, i really don’t want to rewrite flytectl. i’d much prefer it worked

gifted-raincoat-59712

09/15/2022, 1:55 PM

could there be anything in my environment that is throwing things out of whack? hard to imagine, given the only changes between internal and external are • admin endpoint • insecure flag

icy-agent-73298

09/15/2022, 1:56 PM

i agree too. lets wait for Katrina and may be we can setup a working session to debug this .

icy-agent-73298

09/15/2022, 1:57 PM

Also would you mind trying with port-forwarding admin too

gifted-raincoat-59712

09/15/2022, 2:41 PM

ok. port-forwarding from my mac.

Copy code

kubectl port-forward -n flyte deployment/flyteadmin 8081:81
Forwarding from 127.0.0.1:8081 -> 81
Forwarding from [::1]:8081 -> 81

Copy code

flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: only .tar, .tar.gz and .tgz extension archives are supported

and drops the connection to flyteadmin

Copy code

E0915 10:31:57.321401   41651 portforward.go:406] an error occurred forwarding 8081 -> 81: error forwarding port 81 to pod babec7281e7ff3e68177473150b934fb7102177d8b28be37bf6cba7c9d1e4359, uid : failed to execute portforward in network namespace "/var/run/netns/cni-0a96f076-33e0-b2f2-65a8-2b5b2db28fe1": failed to dial 81: dial tcp4 127.0.0.1:81: connect: connection refused
E0915 10:31:57.321751   41651 portforward.go:234] lost connection to pod
Handling connection for 8081
E0915 10:31:57.322387   41651 portforward.go:346] error creating error stream for port 8081 -> 81: EOF

same behavior from get projects command. connection drops.

Copy code

flytectl get projects \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8081: connect: connection refused"

icy-agent-73298

09/15/2022, 2:43 PM

Use the grpc port 8089 for flyteadmin instead of the service port 81

gifted-raincoat-59712

09/15/2022, 2:47 PM

my bad. get projects, same behavior, no output

Copy code

flytectl get projects \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"project.go:89"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:46:48-04:00"}

icy-agent-73298

09/15/2022, 2:48 PM

hmmm ok. and with positional args it works right

gifted-raincoat-59712

09/15/2022, 2:48 PM

Copy code

flytectl get projects \                                                    
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
> foo
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"project.go:102"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:48:20-04:00"}
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

icy-agent-73298

09/15/2022, 2:49 PM

ok. seems problem lies with conversion to proto.Message slice

gifted-raincoat-59712

09/15/2022, 2:51 PM

remember, the real problem is trying to run

register files

Copy code

flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}

unclear if protobufs are also at issue here

gifted-raincoat-59712

09/15/2022, 2:52 PM

but it does appear to be related to input/output variables

gifted-raincoat-59712

09/15/2022, 2:55 PM

https://github.com/flyteorg/flytectl/blob/f18901c734861b440bde6e77b664c54fe3d37255/cmd/register/files.go#L131

icy-agent-73298

09/15/2022, 2:56 PM

Can you pass the archive as the last argument . --archive is actually a boolean flag

icy-agent-73298

09/15/2022, 2:57 PM

Pass this

msat-platform-flyte-package.tgz

towards the end

gifted-raincoat-59712

09/15/2022, 2:58 PM

i was just reading that

Copy code

--archive                       Pass in archive file either an http link or local path.

gifted-raincoat-59712

09/15/2022, 2:59 PM

remember, external endpoint works with same flags. i just ran it again. this works fine.

Copy code

flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///EXTERNAL:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth

icy-agent-73298

09/15/2022, 3:05 PM

hmm . https://github.com/flyteorg/flytectl/blob/master/cmd/config/subcommand/register/files_config.go#L21 If you look at the examples in here https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go and how we are getting the files is basically from arg slice and not the flags values https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go#L129 and hence i suggested to use that format . but yeah it shoudl work this way too . Passing --archive inteprets as true and uses msat-platform-flyte-package.tgz as args[0]

gifted-raincoat-59712

09/15/2022, 3:12 PM

ok. so i could be using the CLI all wrong, but it still works on external. 🤯 let me try your suggestion

icy-agent-73298

09/15/2022, 3:13 PM

it wont matter imo .

gifted-raincoat-59712

09/15/2022, 3:13 PM

correct

Copy code

flytectl register files \                                                  
  --project dpp \
  --domain default \
  --archive \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
  msat-platform-flyte-package.tgz
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:13:01-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}

icy-agent-73298

09/15/2022, 4:01 PM

I tried a package with the same name locally and it worked for me

msat-platform-flyte-package.tgz

Curious how are you running these. Is it directly as command on terminal or you run some script which contains this command

gifted-raincoat-59712

09/15/2022, 5:07 PM

i’m exec’ing into our own pod. can i ask how you are running it? within a pod? and which Docker image? let me try some things to try and eliminate my environment as the culprit.

gifted-raincoat-59712

09/15/2022, 5:08 PM

keep forgetting, i’m seeing the same problem connecting (port-forwarding) from local mac too

acceptable-policeman-57188

09/15/2022, 5:11 PM

hey sorry just catching up - what's the issue here?

gifted-raincoat-59712

09/15/2022, 5:11 PM

https://flyte-org.slack.com/archives/CP2HDHKE1/p1663245404796489?thread_ts=1663199286.385729&cid=CP2HDHKE1

gifted-raincoat-59712

09/15/2022, 5:52 PM

@acceptable-policeman-57188 long story short, I cannot seem to run any

flytectl

commands with success or expected results when I run from INSIDE the kubernetes cluster, hitting the internal flyteadmin endpoint,

flyteadmin.flyte.svc.cluster.local:81

acceptable-policeman-57188

09/15/2022, 6:22 PM

interesting but only with no positional argument?

acceptable-policeman-57188

09/15/2022, 6:23 PM

i don't know that we've seen this before with different behavior based on the endpoint, that is unexpected

gifted-raincoat-59712

09/15/2022, 6:24 PM

well, i’m really trying to run

flytectl register files

, but it fails all the time, because i don’t think it can find the right tarball archive file from the CLI args. this is REALLY what i need to work https://flyte-org.slack.com/archives/CP2HDHKE1/p1663249419061439?thread_ts=1663199286.385729&cid=CP2HDHKE1

gifted-raincoat-59712

09/15/2022, 6:26 PM

i don’t know that we’ve seen this before with different behavior based on the endpoint, that is unexpected

agreed. seems to me that flytectl <> flyteadmin struggles with inputs/outputs when using internal endpoint

acceptable-policeman-57188

09/15/2022, 6:26 PM

wonder if this is some ingress issue?

gifted-raincoat-59712

09/15/2022, 6:27 PM

well, it WORKS when using the ingress, through the external endpoint. it fails when it bypasses the ingress to the internal endpoint

acceptable-policeman-57188

09/15/2022, 6:28 PM

Copy code

"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported"

this seems separate too

gifted-raincoat-59712

09/15/2022, 6:30 PM

i don’t think flytectl picked up the arg.

Copy code

--archive msat-platform-flyte-package.tgz

no idea what it thinks i passed in

acceptable-policeman-57188

09/15/2022, 6:31 PM

but only in the case of hitting the internal endpoint? this is wild, that shouldn't affect how command line flags are passed

icy-agent-73298

09/16/2022, 11:34 AM

I am running from mac terminal and not from inside any pod but it should still be the same . Can you check if there are any control characters in your commandline thats causing it to not read the file name correctly .

icy-agent-73298

09/16/2022, 11:35 AM

I have tried from within flyte sandbox environment by installing flytectl inside minio container and using flyteadmin svc url and i can’t still replicate the issue faced with

get project

Copy code

I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure flyteexamples
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows

183 Views

Open in Slack

Previous Next