Justin Tyberg

    Justin Tyberg

    1 week ago
    I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in
    flyte
    namespace • flytectl running in
    foo
    namespace If I run flytectl from within the pod in
    foo
    namespace, and hit the external endpoint (through the ingress), it works
    bin/flytectl get projects \
      --admin.endpoint dns:///EXTERNAL_FQDN:443 \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    However, if I try to hit the internal grpc endpoint, I get nada. No output.
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    
    echo $?
    0
    🤔
    Samhita Alla

    Samhita Alla

    1 week ago
    @Prafulla Mahindrakar, could you help?
    p

    Prafulla Mahindrakar

    1 week ago
    Hi @Justin Tyberg thats strange, can you also add more logging to the command by passing
    --logger.level 6
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i should have added the logger first thing. seems unauthenticted? odd because it’s the same client secret
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --logger.level 6
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
    p

    Prafulla Mahindrakar

    1 week ago
    These log mention that it attempted and it did retrieve the projects and it contains 1 project.
    {"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:19:14Z"}
    {"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:19:14Z"}
    But it doesn’t print the o/p though . hmm.
    Can you try
    -o yaml
    output format
    The client tries unauthenticated communication first https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L63 and then uses creds flow if it fails https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/auth_interceptor.go#L69 So you can ignore the error log . But the communication did went through in your case and retrieved the data . It strangely failed in last step of printing the data
    Justin Tyberg

    Justin Tyberg

    1 week ago
    yaml output is also empty
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
    p

    Prafulla Mahindrakar

    1 week ago
    hmm strange. Do you mind adding more logging on your end to see what is it returning from admin Also mind checking what happens when you hit admin logshttps://github.com/flyteorg/flytectl/blob/master/cmd/get/project.go#L102-L103
    Justin Tyberg

    Justin Tyberg

    1 week ago
    I’m not really in a position to edit the source to add more logging, if that’s what you’re asking.
    p

    Prafulla Mahindrakar

    1 week ago
    yeah i was hoping we could do that. May be we can try other ways . What happens when you pass in project name
    dpp
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml \
    >.  --logger.level 6 \
    >.  dpp
    Another you could try is portforwarding the admin and using that instead of
    flyteadmin.flyte.svc.cluster.local:81
    Justin Tyberg

    Justin Tyberg

    1 week ago
    same empty output
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml \
    >   --logger.level 6 \
    >   --project dpp
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:58:54Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T11:58:54Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T11:58:54Z"}
    {"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T11:58:54Z"}
    p

    Prafulla Mahindrakar

    1 week ago
    What if you pass in non-existent project
    Justin Tyberg

    Justin Tyberg

    1 week ago
    same
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml \
    >   --logger.level 6 \
    >   --project foo
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:02:55Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:02:55Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:02:55Z"}
    {"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:02:55Z"}
    p

    Prafulla Mahindrakar

    1 week ago
    It expects the arg[0] to be passed as the name of the project. --project flag is ignored in this case
    --project foo
    Can you pass the project the way i sent you
    Justin Tyberg

    Justin Tyberg

    1 week ago
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml \
    >   --logger.level 6 \
    >   dpp
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
    {"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T12:07:50Z"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T12:07:50Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2022-09-15T12:07:50Z"}
    {"json":{},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2022-09-15T12:07:50Z"}
    {"json":{},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T12:07:50Z"}
    description: dpp description
    domains:
    - id: default
      name: default
    id: dpp
    name: dpp
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --output yaml \
    >   foo
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    description: dpp description
    domains:
    - id: default
      name: default
    id: dpp
    name: dpp
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   foo
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    seems a bit odd. passing any argument ends up printing the output
    p

    Prafulla Mahindrakar

    1 week ago
    Strange i dont see this behavior .
    (lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects flyteexamples --logger.level=0
     --------------- --------------- --------------------------- 
    | ID      | NAME     | DESCRIPTION        |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    1 rows
    (lab_project) ➜ flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 
     --------------- --------------- --------------------------- 
    | ID      | NAME     | DESCRIPTION        |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    | flytetester  | flytetester  | flytetester description  |
     --------------- --------------- --------------------------- 
    | flytesnacks  | flytesnacks  | flytesnacks description  |
     --------------- --------------- ---------------------------
    Can you move all flags to you config.yaml file . You can define one using
    flytectl config init --host=flyteadmin.flyte.svc.cluster.local:81
    and update the authType, clientId, clientSecretLocation in that file. Also assuming you have latest flytectl
    Justin Tyberg

    Justin Tyberg

    1 week ago
    does the client use different code paths depending on whether there’s a config file?
    can you try without a config file?
    p

    Prafulla Mahindrakar

    1 week ago
    Same results for me without the config file . There is not much deviation other than reading the flags from the file
    (lab_project) ➜  flytectl git:(device-auth) ✗ mv ~/.flyte/config.yaml ~/.flyte/config.yaml_bkp
    (lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081              
     --------------- --------------- --------------------------- 
    | ID            | NAME          | DESCRIPTION               |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    | flytetester   | flytetester   | flytetester description   |
     --------------- --------------- --------------------------- 
    | flytesnacks   | flytesnacks   | flytesnacks description   |
     --------------- --------------- --------------------------- 
    3 rows
    (lab_project) ➜  flytectl git:(device-auth) ✗ flytectl get projects --logger.level=0 --admin.endpoint dns:///localhost:30081 flyteexamples
     --------------- --------------- --------------------------- 
    | ID            | NAME          | DESCRIPTION               |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    1 rows
    have my config.yaml here
    ~/.flyte/config.yaml
    which i have moved to backup
    Justin Tyberg

    Justin Tyberg

    1 week ago
    to recap where we are external endpoint, without positional argument = OK
    bin/flytectl get projects \
    >   --admin.endpoint dns:///EXTERNAL:443 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    external endpoint, with positional argument = OK
    bin/flytectl get projects \
    >   --admin.endpoint dns:///EXTERNAL:443 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   dpp
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    internal endpoint, without positional argument = FAIL (no output)
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
    internal endpoint, with positional argument = OK
    bin/flytectl get projects \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.insecure true \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   dpp
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    so how does code path differ for internal endpoint without positional argument?
    p

    Prafulla Mahindrakar

    1 week ago
    This one is not working in my opinion correctly since anything you pass is returning you the project
    internal endpoint, with positional argument = OK
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i would agree with that
    Seems in the positional case we dont call
    ProjectToProtoMessages
    That function looks harmless to me as it just create a new proto.Message slice of the same length of projects slice and adds the projects proto
    func ProjectToProtoMessages(l []*admin.Project) []proto.Message {
    	messages := make([]proto.Message, 0, len(l))
    	for _, m := range l {
    		messages = append(messages, m)
    	}
    	return messages
    }
    Justin Tyberg

    Justin Tyberg

    1 week ago
    right, but if that block is not called (no arguments), then we’re not seeing any output, which is the original problem
    p

    Prafulla Mahindrakar

    1 week ago
    yes
    Strange it works well from EXTERNAL endpoint though
    Justin Tyberg

    Justin Tyberg

    1 week ago
    btw, this is my version
    {
      "App": "flytectl",
      "Build": "f18901c",
      "Version": "0.6.13",
      "BuildTime": "2022-09-15 12:53:00.692399756 +0000 UTC m=+0.025439443"
    }
    and flyte helm chart 1.1.0
    p

    Prafulla Mahindrakar

    1 week ago
    those versions look good to me .
    adding @katrina if we have seen such cases before. looks a strange issue
    @Justin Tyberg another thing that could help is any other flyte get also encounters similar issue. eg :
    flytectl get workflows
    Justin Tyberg

    Justin Tyberg

    1 week ago
    here is what i’m really trying to do.
    flytectl register files
    if i use the external endpoint, it SUCCEEDS
    bin/flytectl register files \
    >   --project dpp \
    >   --domain default \
    >   --archive msat-platform-flyte-package.tgz \
    >   --version 7b89d7d \
    >   --force \
    >   --admin.endpoint dns:///EXTERNAL:443 \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    ...
    85 rows
    but if i use internal endpoint, FAIL. here it complains about the tarball not being a tar gzipped file, but it is.
    bin/flytectl register files \
    >   --project dpp \
    >   --domain default \
    >   --archive msat-platform-flyte-package.tgz \
    >   --version 7b89d7d \
    >   --force \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.insecure true \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.clientSecretLocation /etc/secrets/client_secret
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}
    Error: only .tar, .tar.gz and .tgz extension archives are supported
    {"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:32:23Z"}
    
    file msat-platform-flyte-package.tgz
    msat-platform-flyte-package.tgz: gzip compressed data, was "msat-platform-flyte-7b89d7d.tgz", last modified: Thu Sep 15 13:20:16 2022, max compression, original size modulo 2^32 327680
    with more log output
    bin/flytectl register files \
    >   --project dpp \
    >   --domain default \
    >   --archive msat-platform-flyte-package.tgz \
    >   --version 7b89d7d \
    >   --force \
    >   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
    >   --admin.insecure true \
    >   --admin.authType ClientSecret \
    >   --admin.clientId flytepropeller \
    >   --admin.clientSecretLocation /etc/secrets/client_secret \
    >   --logger.level 6
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
    {"json":{},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
    {"json":{},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
    {"json":{},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T13:43:09Z"}
    {"json":{},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T13:43:09Z"}
    {"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T13:43:09Z"}
    {"json":{},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
    Error: only .tar, .tar.gz and .tgz extension archives are supported
    {"json":{},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T13:43:09Z"}
    p

    Prafulla Mahindrakar

    1 week ago
    Can we keep the registration issue on a separate thread . Could be different issue here
    Justin Tyberg

    Justin Tyberg

    1 week ago
    well, that failure led me to run the simplest command i could.
    get projects
    can i just bypass flytectl? run a curl command or something?
    ah. grpc
    p

    Prafulla Mahindrakar

    1 week ago
    yeah . i haven’t tried but postman now supports grpc https://blog.postman.com/postman-now-supports-grpc/
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i mean from a pod CLI.
    Justin Tyberg

    Justin Tyberg

    1 week ago
    then again, i really don’t want to rewrite flytectl. i’d much prefer it worked
    could there be anything in my environment that is throwing things out of whack? hard to imagine, given the only changes between internal and external are • admin endpoint • insecure flag
    p

    Prafulla Mahindrakar

    1 week ago
    i agree too. lets wait for Katrina and may be we can setup a working session to debug this .
    Also would you mind trying with port-forwarding admin too
    Justin Tyberg

    Justin Tyberg

    1 week ago
    ok. port-forwarding from my mac.
    kubectl port-forward -n flyte deployment/flyteadmin 8081:81
    Forwarding from 127.0.0.1:8081 -> 81
    Forwarding from [::1]:8081 -> 81
    register files fails in the same manner
    flytectl register files \
      --project dpp \
      --domain default \
      --archive msat-platform-flyte-package.tgz \
      --version 7b89d7d \
      --force \
      --admin.endpoint dns:///localhost:8081 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth
    Error: only .tar, .tar.gz and .tgz extension archives are supported
    and drops the connection to flyteadmin
    E0915 10:31:57.321401   41651 portforward.go:406] an error occurred forwarding 8081 -> 81: error forwarding port 81 to pod babec7281e7ff3e68177473150b934fb7102177d8b28be37bf6cba7c9d1e4359, uid : failed to execute portforward in network namespace "/var/run/netns/cni-0a96f076-33e0-b2f2-65a8-2b5b2db28fe1": failed to dial 81: dial tcp4 127.0.0.1:81: connect: connection refused
    E0915 10:31:57.321751   41651 portforward.go:234] lost connection to pod
    Handling connection for 8081
    E0915 10:31:57.322387   41651 portforward.go:346] error creating error stream for port 8081 -> 81: EOF
    same behavior from get projects command. connection drops.
    flytectl get projects \
      --admin.endpoint dns:///localhost:8081 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth
    Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8081: connect: connection refused"
    p

    Prafulla Mahindrakar

    1 week ago
    Use the grpc port 8089 for flyteadmin instead of the service port 81
    Justin Tyberg

    Justin Tyberg

    1 week ago
    my bad. get projects, same behavior, no output
    flytectl get projects \
      --admin.endpoint dns:///localhost:8089 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth \
      --logger.level 6
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:46:48-04:00"}
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:46:48-04:00"}
    {"json":{"src":"project.go:89"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:46:48-04:00"}
    p

    Prafulla Mahindrakar

    1 week ago
    hmmm ok. and with positional args it works right
    Justin Tyberg

    Justin Tyberg

    1 week ago
    flytectl get projects \                                                    
      --admin.endpoint dns:///localhost:8089 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth \
      --logger.level 6 \
    > foo
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
    {"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-09-15T10:48:20-04:00"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [files] updated. No update handler registered.","ts":"2022-09-15T10:48:20-04:00"}
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:48:20-04:00"}
    {"json":{"src":"project.go:102"},"level":"debug","msg":"Retrieved 1 projects","ts":"2022-09-15T10:48:20-04:00"}
     ----- ------ ----------------- 
    | ID  | NAME | DESCRIPTION     |
     ----- ------ ----------------- 
    | dpp | dpp  | dpp description |
     ----- ------ ----------------- 
    1 rows
    p

    Prafulla Mahindrakar

    1 week ago
    ok. seems problem lies with conversion to proto.Message slice
    Justin Tyberg

    Justin Tyberg

    1 week ago
    remember, the real problem is trying to run
    register files
    .
    flytectl register files \
      --project dpp \
      --domain default \
      --archive msat-platform-flyte-package.tgz \
      --version 7b89d7d \
      --force \
      --admin.endpoint dns:///localhost:8089 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth \
      --logger.level 6
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-09-15T10:51:10-04:00"}
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T10:51:10-04:00"}
    {"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
    Error: only .tar, .tar.gz and .tgz extension archives are supported
    {"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T10:51:10-04:00"}
    unclear if protobufs are also at issue here
    but it does appear to be related to input/output variables
    p

    Prafulla Mahindrakar

    1 week ago
    Can you pass the archive as the last argument . --archive is actually a boolean flag
    Pass this
    msat-platform-flyte-package.tgz
    towards the end
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i was just reading that
    --archive                       Pass in archive file either an http link or local path.
    remember, external endpoint works with same flags. i just ran it again. this works fine.
    flytectl register files \
      --project dpp \
      --domain default \
      --archive msat-platform-flyte-package.tgz \
      --version 7b89d7d \
      --force \
      --admin.endpoint dns:///EXTERNAL:443 \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth
    p

    Prafulla Mahindrakar

    1 week ago
    hmm . https://github.com/flyteorg/flytectl/blob/master/cmd/config/subcommand/register/files_config.go#L21 If you look at the examples in here https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go and how we are getting the files is basically from arg slice and not the flags values https://github.com/flyteorg/flytectl/blob/master/cmd/register/files.go#L129 and hence i suggested to use that format . but yeah it shoudl work this way too . Passing --archive inteprets as true and uses msat-platform-flyte-package.tgz as args[0]
    Justin Tyberg

    Justin Tyberg

    1 week ago
    ok. so i could be using the CLI all wrong, but it still works on external. 🤯 let me try your suggestion
    p

    Prafulla Mahindrakar

    1 week ago
    it wont matter imo .
    Justin Tyberg

    Justin Tyberg

    1 week ago
    correct
    flytectl register files \                                                  
      --project dpp \
      --domain default \
      --archive \
      --version 7b89d7d \
      --force \
      --admin.endpoint dns:///localhost:8089 \
      --admin.insecure true \
      --admin.authType ClientSecret \
      --admin.clientId flytepropeller \
      --admin.clientSecretLocation /tmp/flyte-secret-auth \
      --logger.level 6 \
      msat-platform-flyte-package.tgz
    {"json":{"src":"client.go:64"},"level":"info","msg":"Initialized Admin client","ts":"2022-09-15T11:13:01-04:00"}
    {"json":{"src":"files.go:131"},"level":"error","msg":"error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
    Error: only .tar, .tar.gz and .tgz extension archives are supported
    {"json":{"src":"main.go:13"},"level":"error","msg":"only .tar, .tar.gz and .tgz extension archives are supported","ts":"2022-09-15T11:13:01-04:00"}
    p

    Prafulla Mahindrakar

    1 week ago
    I tried a package with the same name locally and it worked for me
    msat-platform-flyte-package.tgz
    Curious how are you running these. Is it directly as command on terminal or you run some script which contains this command
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i’m exec’ing into our own pod. can i ask how you are running it? within a pod? and which Docker image? let me try some things to try and eliminate my environment as the culprit.
    keep forgetting, i’m seeing the same problem connecting (port-forwarding) from local mac too
    k

    katrina

    1 week ago
    hey sorry just catching up - what's the issue here?
    @katrina long story short, I cannot seem to run any
    flytectl
    commands with success or expected results when I run from INSIDE the kubernetes cluster, hitting the internal flyteadmin endpoint,
    flyteadmin.flyte.svc.cluster.local:81
    k

    katrina

    1 week ago
    interesting but only with no positional argument?
    i don't know that we've seen this before with different behavior based on the endpoint, that is unexpected
    Justin Tyberg

    Justin Tyberg

    1 week ago
    well, i’m really trying to run
    flytectl register files
    , but it fails all the time, because i don’t think it can find the right tarball archive file from the CLI args. this is REALLY what i need to work https://flyte-org.slack.com/archives/CP2HDHKE1/p1663249419061439?thread_ts=1663199286.385729&cid=CP2HDHKE1
    i don’t know that we’ve seen this before with different behavior based on the endpoint, that is unexpected
    agreed. seems to me that flytectl <> flyteadmin struggles with inputs/outputs when using internal endpoint
    k

    katrina

    1 week ago
    wonder if this is some ingress issue?
    Justin Tyberg

    Justin Tyberg

    1 week ago
    well, it WORKS when using the ingress, through the external endpoint. it fails when it bypasses the ingress to the internal endpoint
    k

    katrina

    1 week ago
    "error while un-archiving files in tmp dir due to only .tar, .tar.gz and .tgz extension archives are supported"
    this seems separate too
    Justin Tyberg

    Justin Tyberg

    1 week ago
    i don’t think flytectl picked up the arg.
    --archive msat-platform-flyte-package.tgz
    no idea what it thinks i passed in
    k

    katrina

    1 week ago
    but only in the case of hitting the internal endpoint? this is wild, that shouldn't affect how command line flags are passed
    p

    Prafulla Mahindrakar

    1 week ago
    I am running from mac terminal and not from inside any pod but it should still be the same . Can you check if there are any control characters in your commandline thats causing it to not read the file name correctly .
    I have tried from within flyte sandbox environment by installing flytectl inside minio container and using flyteadmin svc url and i can’t still replicate the issue faced with
    get project
    .
    I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     --------------- --------------- --------------------------- 
    | ID            | NAME          | DESCRIPTION               |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    | flytetester   | flytetester   | flytetester description   |
     --------------- --------------- --------------------------- 
    | flytesnacks   | flytesnacks   | flytesnacks description   |
     --------------- --------------- --------------------------- 
    3 rows
    I have no name!@minio-68744577b6-kmfrn:/opt/bitnami/minio-client$ flytectl get projects --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81  --admin.insecure flyteexamples
    INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
     --------------- --------------- --------------------------- 
    | ID            | NAME          | DESCRIPTION               |
     --------------- --------------- --------------------------- 
    | flyteexamples | flyteexamples | flyteexamples description |
     --------------- --------------- --------------------------- 
    1 rows