https://flyte.org logo
Join Slack
Powered by
I’m having trouble accessing flyteadmin from flyte...
# ask-the-community
j
I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in 
flyte
namespace • flytectl running in 
foo
namespace If I run flytectl from within the pod in 
foo
namespace, and hit the external endpoint (through the ingress), it works
Copy code
bin/flytectl get projects \
  --admin.endpoint dns:///EXTERNAL_FQDN:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
However, if I try to hit the internal grpc endpoint, I get nada. No output.
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 

echo $?
0
🤔
s
@Prafulla Mahindrakar, could you help?
p
Hi @Justin Tyberg thats strange, can you also add more logging to the command by passing 
--logger.level 6
j
i should have added the logger first thing. seems unauthenticted? odd because it’s the same client secret
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
p
These log mention that it attempted and it did retrieve the projects and it contains 1 project.
Copy code
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
But it doesn’t print the o/p though . hmm.
Can you try 
-o yaml
output format
The client tries unauthenticated communication first https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L63 and then uses creds flow if it fails https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L69 So you can ignore the error log . But the communication did went through in your case and retrieved the data . It strangely failed in last step of printing the data
j
yaml output is also empty
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
p
hmm strange. Do you mind adding more logging on your end to see what is it returning from admin Also mind checking what happens when you hit admin logs https://github.com/flyteorg/flytectl/blob/master/cmd/get/project.go#L102-L103
j
I’m not really in a position to edit the source to add more logging, if that’s what you’re asking.
p
yeah i was hoping we could do that. May be we can try other ways . What happens when you pass in project name 
dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>.  --logger.level 6 \
>.  dpp
Another you could try is portforwarding the admin and using that instead of 
flyteadmin.flyte.svc.cluster.local:81
j
same empty output
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:58:54Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:58:54Z"}
p
What if you pass in non-existent project
j
same
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   --project foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:02:55Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:02:55Z"}
p
It expects the arg[0] to be passed as the name of the project. --project flag is ignored in this case
Copy code
--project foo
Can you pass the project the way i sent you
j
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   --logger.level 6 \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:07:50Z"}
{"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:07:50Z"}
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --output yaml \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
description: dpp description
domains:
- id: default
  name: default
id: dpp
name: dpp
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   foo
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
seems a bit odd. passing any argument ends up printing the output
p
Strange i dont see this behavior .
Copy code
(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects flyteexamples --logger.level=0
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
(lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 
 --------------- --------------- --------------------------- 
| ID      | NAME     | DESCRIPTION        |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester  | flytetester  | flytetester description  |
 --------------- --------------- --------------------------- 
| flytesnacks  | flytesnacks  | flytesnacks description  |
 --------------- --------------- ---------------------------
Can you move all flags to you config.yaml file . You can define one using 
flytectl config init --host=flyteadmin.flyte.svc.cluster.local:81
and update the authType, clientId, clientSecretLocation in that file. Also assuming you have latest flytectl
j
does the client use different code paths depending on whether there’s a config file?
can you try without a config file?
p
Same results for me without the config file . There is not much deviation other than reading the flags from the file
Copy code
(lab_project) ➜  flytectl git:(device-auth) ✗ mv ~/.flyte/config.yaml ~/.flyte/config.yaml_bkp
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081              
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
(lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081 flyteexamples
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
have my config.yaml here 
~/.flyte/config.yaml
which i have moved to backup
j
to recap where we are external endpoint, without positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
external endpoint, with positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
internal endpoint, without positional argument = FAIL (no output)
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
internal endpoint, with positional argument = OK
Copy code
bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   dpp
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
so how does code path differ for internal endpoint without positional argument?
p
This one is not working in my opinion correctly since anything you pass is returning you the project
Copy code
internal endpoint, with positional argument = OK
j
i would agree with that
p
Seems in the positional case we dont call 
ProjectToProtoMessages
That function looks harmless to me as it just create a new proto.Message slice of the same length of projects slice and adds the projects proto
Copy code
func ProjectToProtoMessages(l []*admin.Project) []proto.Message {
	messages := make([]proto.Message, 0, len(l))
	for _, m := range l {
		messages = append(messages, m)
	}
	return messages
}
j
right, but if that block is not called (no arguments), then we’re not seeing any output, which is the original problem
p
yes
Strange it works well from EXTERNAL endpoint though
j
btw, this is my version
Copy code
{
  "App": "flytectl",
  "Build": "f18901c",
  "Version": "0.6.13",
  "BuildTime": "2022-09-15 12:53:00.692399756 +0000 UTC m=+0.025439443"
}
and flyte helm chart 1.1.0
p
those versions look good to me .
adding @katrina if we have seen such cases before. looks a strange issue
@Justin Tyberg another thing that could help is any other flyte get also encounters similar issue. eg : 
flytectl get workflows
j
here is what i’m really trying to do. 
flytectl register files
if i use the external endpoint, it SUCCEEDS
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///EXTERNAL:443 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
...
85 rows
but if i use internal endpoint, FAIL. here it complains about the tarball not being a tar gzipped file, but it is.
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}

file msat-platform-flyte-package.tgz
msat-platform-flyte-package.tgz: gzip compressed data, was "msat-platform-flyte-7b89d7d.tgz", last modified: Thu Sep 15 13:20:16 2022, max compression, original size modulo 2^32 327680
with more log output
Copy code
bin/flytectl register files \
>   --project dpp \
>   --domain default \
>   --archive msat-platform-flyte-package.tgz \
>   --version 7b89d7d \
>   --force \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.insecure true \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.clientSecretLocation /etc/secrets/client_secret \
>   --logger.level 6
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T13:43:09Z"}
{"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
p
Can we keep the registration issue on a separate thread . Could be different issue here
j
well, that failure led me to run the simplest command i could. 
get projects
can i just bypass flytectl? run a curl command or something?
ah. grpc
p
yeah . i haven’t tried but postman now supports grpc https://blog.postman.com/postman-now-supports-grpc/
j
i mean from a pod CLI.
p
j
then again, i really don’t want to rewrite flytectl. i’d much prefer it worked
could there be anything in my environment that is throwing things out of whack? hard to imagine, given the only changes between internal and external are • admin endpoint • insecure flag
p
i agree too. lets wait for Katrina and may be we can setup a working session to debug this .
Also would you mind trying with port-forwarding admin too
j
ok. port-forwarding from my mac.
Copy code
kubectl port-forward -n flyte deployment/flyteadmin 8081:81
Forwarding from 127.0.0.1:8081 -> 81
Forwarding from [::1]:8081 -> 81
register files fails in the same manner
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: only .tar, .tar.gz and .tgz extension archives are supported
and drops the connection to flyteadmin
Copy code
E0915 10:31:57.321401   41651 portforward.go:406] an error occurred forwarding 8081 -> 81: error forwarding port 81 to pod babec7281e7ff3e68177473150b934fb7102177d8b28be37bf6cba7c9d1e4359, uid : failed to execute portforward in network namespace "/var/run/netns/cni-0a96f076-33e0-b2f2-65a8-2b5b2db28fe1": failed to dial 81: dial tcp4 127.0.0.1:81: connect: connection refused
E0915 10:31:57.321751   41651 portforward.go:234] lost connection to pod
Handling connection for 8081
E0915 10:31:57.322387   41651 portforward.go:346] error creating error stream for port 8081 -> 81: EOF
same behavior from get projects command. connection drops.
Copy code
flytectl get projects \
  --admin.endpoint dns:///localhost:8081 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8081: connect: connection refused"
p
Use the grpc port 8089 for flyteadmin instead of the service port 81
j
my bad. get projects, same behavior, no output
Copy code
flytectl get projects \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:46:48-04:00"}
{"json":{"src":"project.go:89"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:46:48-04:00"}
p
hmmm ok. and with positional args it works right
j
Copy code
flytectl get projects \                                                    
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
> foo
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:48:20-04:00"}
{"json":{"src":"project.go:102"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:48:20-04:00"}
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows
p
ok. seems problem lies with conversion to proto.Message slice
j
remember, the real problem is trying to run 
register files
.
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:51:10-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
unclear if protobufs are also at issue here
but it does appear to be related to input/output variables
p
Can you pass the archive as the last argument . --archive is actually a boolean flag
Pass this 
msat-platform-flyte-package.tgz
towards the end
j
i was just reading that
Copy code
--archive                       Pass in archive file either an http link or local path.
remember, external endpoint works with same flags. i just ran it again. this works fine.
Copy code
flytectl register files \
  --project dpp \
  --domain default \
  --archive msat-platform-flyte-package.tgz \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///EXTERNAL:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth
p
hmm . https://github.com/flyteorg/flytectl/blob/master/cmd/config/subcommand/register/files_config.go#L21 If you look at the examples in here https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go and how we are getting the files is basically from arg slice and not the flags values https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go#L129 and hence i suggested to use that format . but yeah it shoudl work this way too . Passing --archive inteprets as true and uses msat-platform-flyte-package.tgz as args[0]
j
ok. so i could be using the CLI all wrong, but it still works on external. 🤯 let me try your suggestion
p
it wont matter imo .
j
correct
Copy code
flytectl register files \                                                  
  --project dpp \
  --domain default \
  --archive \
  --version 7b89d7d \
  --force \
  --admin.endpoint dns:///localhost:8089 \
  --admin.insecure true \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /tmp/flyte-secret-auth \
  --logger.level 6 \
  msat-platform-flyte-package.tgz
{"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:13:01-04:00"}
{"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
Error: only .tar, .tar.gz and .tgz extension archives are supported
{"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
p
I tried a package with the same name locally and it worked for me 
msat-platform-flyte-package.tgz
Curious how are you running these. Is it directly as command on terminal or you run some script which contains this command
j
i’m exec’ing into our own pod. can i ask how you are running it? within a pod? and which Docker image? let me try some things to try and eliminate my environment as the culprit.
keep forgetting, i’m seeing the same problem connecting (port-forwarding) from local mac too
k
hey sorry just catching up - what's the issue here?
j
@katrina long story short, I cannot seem to run any 
flytectl
commands with success or expected results when I run from INSIDE the kubernetes cluster, hitting the internal flyteadmin endpoint, 
flyteadmin.flyte.svc.cluster.local:81
k
interesting but only with no positional argument?
i don't know that we've seen this before with different behavior based on the endpoint, that is unexpected
j
well, i’m really trying to run 
flytectl register files
, but it fails all the time, because i don’t think it can find the right tarball archive file from the CLI args. this is REALLY what i need to work https://flyte-org.slack.com/archives/CP2HDHKE1/p1663249419061439?thread_ts=1663199286.385729&cid=CP2HDHKE1
i don’t know that we’ve seen this before with different behavior based on the endpoint, that is unexpected
agreed. seems to me that flytectl <> flyteadmin struggles with inputs/outputs when using internal endpoint
k
wonder if this is some ingress issue?
j
well, it WORKS when using the ingress, through the external endpoint. it fails when it bypasses the ingress to the internal endpoint
k
Copy code
"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported"
this seems separate too
j
i don’t think flytectl picked up the arg.
Copy code
--archive msat-platform-flyte-package.tgz
no idea what it thinks i passed in
k
but only in the case of hitting the internal endpoint? this is wild, that shouldn't affect how command line flags are passed
p
I am running from mac terminal and not from inside any pod but it should still be the same . Can you check if there are any control characters in your commandline thats causing it to not read the file name correctly .
I have tried from within flyte sandbox environment by installing flytectl inside minio container and using flyteadmin svc url and i can’t still replicate the issue faced with 
get project
.
Copy code
I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
| flytetester   | flytetester   | flytetester description   |
 --------------- --------------- --------------------------- 
| flytesnacks   | flytesnacks   | flytesnacks description   |
 --------------- --------------- --------------------------- 
3 rows
I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure flyteexamples
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 --------------- --------------- --------------------------- 
| ID            | NAME          | DESCRIPTION               |
 --------------- --------------- --------------------------- 
| flyteexamples | flyteexamples | flyteexamples description |
 --------------- --------------- --------------------------- 
1 rows
171 Views
Open in Slack
PreviousNext
Powered by