https://flyte.org logo
Powered by
j

Justin Tyberg

09/14/2022, 11:48 PM
I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in 
flyte
namespace • flytectl running in 
foo
namespace If I run flytectl from within the pod in 
foo
namespace, and hit the external endpoint (through the ingress), it works
Copy code
bin/flytectl get projects \
  --admin.endpoint dns:///EXTERNAL_FQDN:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
However, if I try to hit the internal grpc endpoint, I get nada. No output.
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 

echo $?
0
🤔
s

Samhita Alla

09/15/2022, 9:51 AM
@Prafulla Mahindrakar, could you help?
p

Prafulla Mahindrakar

09/15/2022, 9:59 AM
Hi @Justin Tyberg thats strange, can you also add more logging to the command by passing 
--logger.level 6
j

Justin Tyberg

09/15/2022, 11:21 AM
i should have added the logger first thing. seems unauthenticted? odd because it’s the same client secret
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
p

Prafulla Mahindrakar

09/15/2022, 11:24 AM
These log mention that it attempted and it did retrieve the projects and it contains 1 project.
Copy code
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
But it doesn’t print the o/p though . hmm.
Can you try 
-o yaml
output format
The client tries unauthenticated communication first https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L63 and then uses creds flow if it fails https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L69 So you can ignore the error log . But the communication did went through in your case and retrieved the data . It strangely failed in last step of printing the data
j

Justin Tyberg

09/15/2022, 11:35 AM
yaml output is also empty
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
p

Prafulla Mahindrakar

09/15/2022, 11:38 AM
hmm strange. Do you mind adding more logging on your end to see what is it returning from admin Also mind checking what happens when you hit admin logs https://github.com/flyteorg/flytectl/blob/master/cmd/get/project.go#L102-L103
j

Justin Tyberg

09/15/2022, 11:49 AM
I’m not really in a position to edit the source to add more logging, if that’s what you’re asking.
p

Prafulla Mahindrakar

09/15/2022, 11:56 AM
yeah i was hoping we could do that. May be we can try other ways . What happens when you pass in project name 
dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>.  --logger.level 6 \
>.  dpp
Another you could try is portforwarding the admin and using that instead of 
flyteadmin.flyte.svc.cluster.local:81
j

Justin Tyberg

09/15/2022, 11:59 AM
same empty output
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:58:54Z"}
p

Prafulla Mahindrakar

09/15/2022, 12:00 PM
What if you pass in non-existent project
j

Justin Tyberg

09/15/2022, 12:03 PM
same
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:02:55Z"}
p

Prafulla Mahindrakar

09/15/2022, 12:04 PM
It expects the arg[0] to be passed as the name of the project. --project flag is ignored in this case
Copy code
--project foo
Can you pass the project the way i sent you
j

Justin Tyberg

09/15/2022, 12:08 PM
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:07:50Z"}
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
seems a bit odd. passing any argument ends up printing the output
p

Prafulla Mahindrakar

09/15/2022, 12:18 PM
Strange i dont see this behavior .
Copy code
(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects flyteexamples --logger.level=0
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester  | flytetester  | flytetester description  |
 --------------- --------------- --------------------------- 
| flytesnacks  | flytesnacks  | flytesnacks description  |
 --------------- --------------- ---------------------------
Can you move all flags to you config.yaml file . You can define one using 
flytectl config init --host=flyteadmin.flyte.svc.cluster.local:81
and update the authType, clientId, clientSecretLocation in that file. Also assuming you have latest flytectl
j

Justin Tyberg

09/15/2022, 12:26 PM
does the client use different code paths depending on whether there’s a config file?
can you try without a config file?
p

Prafulla Mahindrakar

09/15/2022, 12:30 PM
Same results for me without the config file . There is not much deviation other than reading the flags from the file
Copy code
(lab_project) ➜  flytectl git:(device-auth) ✗ mv ~/.flyte/config.yaml ~/.flyte/config.yaml_bkp
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081              
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081 flyteexamples
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
have my config.yaml here 
~/.flyte/config.yaml
which i have moved to backup
j

Justin Tyberg

09/15/2022, 12:36 PM
to recap where we are external endpoint, without positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
external endpoint, with positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
internal endpoint, without positional argument = FAIL (no output)
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
internal endpoint, with positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
so how does code path differ for internal endpoint without positional argument?
p

Prafulla Mahindrakar

09/15/2022, 12:37 PM
This one is not working in my opinion correctly since anything you pass is returning you the project
Copy code
internal endpoint, with positional argument = OK
j

Justin Tyberg

09/15/2022, 12:38 PM
i would agree with that
p

Prafulla Mahindrakar

09/15/2022, 12:40 PM
Seems in the positional case we dont call 
ProjectToProtoMessages
That function looks harmless to me as it just create a new proto.Message slice of the same length of projects slice and adds the projects proto
Copy code
func ProjectToProtoMessages(l []*admin.Project) []proto.Message {
	messages := make([]proto.Message, 0, len(l))
	for _, m := range l {
		messages = append(messages, m)
	}
	return messages
}
j

Justin Tyberg

09/15/2022, 12:48 PM
right, but if that block is not called (no arguments), then we’re not seeing any output, which is the original problem
p

Prafulla Mahindrakar

09/15/2022, 12:50 PM
yes
Strange it works well from EXTERNAL endpoint though
j

Justin Tyberg

09/15/2022, 12:53 PM
btw, this is my version
Copy code
{
  "App": "flytectl",
  "Build": "f18901c",
  "Version": "0.6.13",
  "BuildTime": "2022-09-15 12:53:00.692399756 +0000 UTC m=+0.025439443"
}
and flyte helm chart 1.1.0
p

Prafulla Mahindrakar

09/15/2022, 12:57 PM
those versions look good to me .
adding @katrina if we have seen such cases before. looks a strange issue
@Justin Tyberg another thing that could help is any other flyte get also encounters similar issue. eg : 
flytectl get workflows
j

Justin Tyberg

09/15/2022, 1:39 PM
here is what i’m really trying to do. 
flytectl register files
if i use the external endpoint, it SUCCEEDS
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
...
85 rows
but if i use internal endpoint, FAIL. here it complains about the tarball not being a tar gzipped file, but it is.
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}

file msat-platform-flyte-package.tgz
msat-platform-flyte-package.tgz: gzip compressed data, was "msat-platform-flyte-7b89d7d.tgz", last modified: Thu Sep 15 13:20:16 2022, max compression, original size modulo 2^32 327680
with more log output
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
p

Prafulla Mahindrakar

09/15/2022, 1:47 PM
Can we keep the registration issue on a separate thread . Could be different issue here
j

Justin Tyberg

09/15/2022, 1:47 PM
well, that failure led me to run the simplest command i could. 
get projects
can i just bypass flytectl? run a curl command or something?
ah. grpc
p

Prafulla Mahindrakar

09/15/2022, 1:49 PM
yeah . i haven’t tried but postman now supports grpc https://blog.postman.com/postman-now-supports-grpc/
j

Justin Tyberg

09/15/2022, 1:51 PM
i mean from a pod CLI.
p

Prafulla Mahindrakar

09/15/2022, 1:53 PM
j

Justin Tyberg

09/15/2022, 1:54 PM
then again, i really don’t want to rewrite flytectl. i’d much prefer it worked
could there be anything in my environment that is throwing things out of whack? hard to imagine, given the only changes between internal and external are • admin endpoint • insecure flag
p

Prafulla Mahindrakar

09/15/2022, 1:56 PM
i agree too. lets wait for Katrina and may be we can setup a working session to debug this .
Also would you mind trying with port-forwarding admin too
j

Justin Tyberg

09/15/2022, 2:41 PM
ok. port-forwarding from my mac.
Copy code
kubectl port-forward -n flyte deployment/flyteadmin 8081:81
Forwarding from 127.0.0.1:8081 -> 81
Forwarding from [::1]:8081 -> 81
register files fails in the same manner
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: only .tar, .tar.gz and .tgz extension archives are supported
and drops the connection to flyteadmin
Copy code
E0915 10:31:57.321401   41651 portforward.go:406] an error occurred forwarding 8081 -> 81: error forwarding port 81 to pod babec7281e7ff3e68177473150b934fb7102177d8b28be37bf6cba7c9d1e4359, uid : failed to execute portforward in network namespace "/var/run/netns/cni-0a96f076-33e0-b2f2-65a8-2b5b2db28fe1": failed to dial 81: dial tcp4 127.0.0.1:81: connect: connection refused
E0915 10:31:57.321751   41651 portforward.go:234] lost connection to pod
Handling connection for 8081
E0915 10:31:57.322387   41651 portforward.go:346] error creating error stream for port 8081 -> 81: EOF
same behavior from get projects command. connection drops.
Copy code
flytectl get projects \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8081: connect: connection refused"
p

Prafulla Mahindrakar

09/15/2022, 2:43 PM
Use the grpc port 8089 for flyteadmin instead of the service port 81
j

Justin Tyberg

09/15/2022, 2:47 PM
my bad. get projects, same behavior, no output
Copy code
flytectl get projects \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"project.go:89"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:46:48-04:00"}
p

Prafulla Mahindrakar

09/15/2022, 2:48 PM
hmmm ok. and with positional args it works right
j

Justin Tyberg

09/15/2022, 2:48 PM
Copy code
flytectl get projects \                                                    
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
> foo
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"project.go:102"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:48:20-04:00"}
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
p

Prafulla Mahindrakar

09/15/2022, 2:49 PM
ok. seems problem lies with conversion to proto.Message slice
j

Justin Tyberg

09/15/2022, 2:51 PM
remember, the real problem is trying to run 
register files
.
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
unclear if protobufs are also at issue here
but it does appear to be related to input/output variables
p

Prafulla Mahindrakar

09/15/2022, 2:56 PM
Can you pass the archive as the last argument . --archive is actually a boolean flag
Pass this 
msat-platform-flyte-package.tgz
towards the end
j

Justin Tyberg

09/15/2022, 2:58 PM
i was just reading that
Copy code
--archive                       Pass in archive file either an http link or local path.
remember, external endpoint works with same flags. i just ran it again. this works fine.
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///EXTERNAL:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
p

Prafulla Mahindrakar

09/15/2022, 3:05 PM
hmm . https://github.com/flyteorg/flytectl/blob/master/cmd/config/subcommand/register/files_config.go#L21 If you look at the examples in here https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go and how we are getting the files is basically from arg slice and not the flags values https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go#L129 and hence i suggested to use that format . but yeah it shoudl work this way too . Passing --archive inteprets as true and uses msat-platform-flyte-package.tgz as args[0]
j

Justin Tyberg

09/15/2022, 3:12 PM
ok. so i could be using the CLI all wrong, but it still works on external. 🤯 let me try your suggestion
p

Prafulla Mahindrakar

09/15/2022, 3:13 PM
it wont matter imo .
j

Justin Tyberg

09/15/2022, 3:13 PM
correct
Copy code
flytectl register files \                                                  
  --project dpp \
  --domain default \
  --archive \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
  msat-platform-flyte-package.tgz
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:13:01-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
p

Prafulla Mahindrakar

09/15/2022, 4:01 PM
I tried a package with the same name locally and it worked for me 
msat-platform-flyte-package.tgz
Curious how are you running these. Is it directly as command on terminal or you run some script which contains this command
j

Justin Tyberg

09/15/2022, 5:07 PM
i’m exec’ing into our own pod. can i ask how you are running it? within a pod? and which Docker image? let me try some things to try and eliminate my environment as the culprit.
keep forgetting, i’m seeing the same problem connecting (port-forwarding) from local mac too
k

katrina

09/15/2022, 5:11 PM
hey sorry just catching up - what's the issue here?
j

Justin Tyberg

09/15/2022, 5:11 PM
@katrina long story short, I cannot seem to run any 
flytectl
commands with success or expected results when I run from INSIDE the kubernetes cluster, hitting the internal flyteadmin endpoint, 
flyteadmin.flyte.svc.cluster.local:81
k

katrina

09/15/2022, 6:22 PM
interesting but only with no positional argument?
i don't know that we've seen this before with different behavior based on the endpoint, that is unexpected
j

Justin Tyberg

09/15/2022, 6:24 PM
well, i’m really trying to run 
flytectl register files
, but it fails all the time, because i don’t think it can find the right tarball archive file from the CLI args. this is REALLY what i need to work https://flyte-org.slack.com/archives/CP2HDHKE1/p1663249419061439?thread_ts=1663199286.385729&cid=CP2HDHKE1
i don’t know that we’ve seen this before with different behavior based on the endpoint, that is unexpected
agreed. seems to me that flytectl <> flyteadmin struggles with inputs/outputs when using internal endpoint
k

katrina

09/15/2022, 6:26 PM
wonder if this is some ingress issue?
j

Justin Tyberg

09/15/2022, 6:27 PM
well, it WORKS when using the ingress, through the external endpoint. it fails when it bypasses the ingress to the internal endpoint
k

katrina

09/15/2022, 6:28 PM
Copy code
"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported"
this seems separate too
j

Justin Tyberg

09/15/2022, 6:30 PM
i don’t think flytectl picked up the arg.
Copy code
--archive msat-platform-flyte-package.tgz
no idea what it thinks i passed in
k

katrina

09/15/2022, 6:31 PM
but only in the case of hitting the internal endpoint? this is wild, that shouldn't affect how command line flags are passed
p

Prafulla Mahindrakar

09/16/2022, 11:34 AM
I am running from mac terminal and not from inside any pod but it should still be the same . Can you check if there are any control characters in your commandline thats causing it to not read the file name correctly .
I have tried from within flyte sandbox environment by installing flytectl inside minio container and using flyteadmin svc url and i can’t still replicate the issue faced with 
get project
.
Copy code
I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure flyteexamples
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
8 Views
Powered by