Hi Flyte Fellows, I have a question about Flytectl...
# flyte-deployment
s
Hi Flyte Fellows, I have a question about Flytectl Auth. There are 2 major concerns when testing flytectl auth over Pkce which I will summarize in 🧵 :
1. The way the flytectl binaries available in the upstream repo are built makes them not use MacOS’s system name resolver, so they will not send the DNS request for flyte deployment url through our VPN. A workaround is to copy the contents of
/etc/resolver/company_vpn_resolver
and paste it as the first nameserver in
/etc/resolve.conf
. Building the
flytectl
binary ourselves on a MacOS (not cross-compiling) should allow it to use the system name resolver. Have people in flyte used
flytectl
and auth on MacOS? Hoping to learn more about there experiences.
2. Authenticating
flytectl
in a container has two problems: attempting to open a browser and storing the secret locally. • Browser Problem: flytctl in a container tries to open a browser with
xdg-open
. This fails. Ideally
flytectl
would have an option to only print the URI and not try to open it. Is it worthwhile to create a github issue for flytectl to output a URL for authentication instead of trying to open the URL ?
flytectl
running in containers would really benefit from this by mounting a browser port onto the container and opening the URL by just copying and pasting the URL • Storing Local Secret: The secret produced during authentication needs to be stored somewhere.
flytectl
stores the key using GitHub - zalando/go-keyring: Cross-platform keyring interface for Go which on linux relies on dbus and GNOME keyring. I’m unsure if we should install GNOME keyring itself in the container or if we can implement the parts of the Secret Service API that flytectl needs in a python program and store the secret only in memory, not in a file.
Above 3 problems are right now major issues while trying to integrate Flyte into our company's Auth system and create fully authenticated deployments. Please let me know what are your thoughts or if it is worthwhile to create Github tickets associated with above problems.
k
Copy code
Browser Problem: flytctl in a container tries to open a browser with xdg-open. This fails. Ideally flytectl would have an option to only print the URI and not try to open it. Is it worthwhile to create a github issue for flytectl to output a URL for authentication instead of trying to open the URL ? flytectl running in containers would really benefit from this by mounting a browser port onto the container and opening the URL by just copying and pasting the URL
@Shahwar Saleem we recommend you use the
client secrets
flow instead of browser based authentication
. Have people in flyte used
flytectl
and auth on MacOS
We use this all the time, infact just used it. It works fine.
cc @Prafulla Mahindrakar / @Yuvraj for examples
p
• Can you expand more on Problem 1 . How are you installing the flytectl binary. Do you have to change dns settings to just do these steps ◦ brew install flyteorg/homebrew-tap/flytectl ◦ OR curl -sL https://ctl.flyte.org/install | bash OR curl -s https://raw.githubusercontent.com/flyteorg/flytectl/master/install.sh |bash • Problem 2 ◦ part 1: Browser issue : You can use client_secret method of auth to not use the default method which uses browser based authentication .We use the following way in our CI by reusing the flypropeller secret https://github.com/flyteorg/flytetools/blob/master/functional-tests/config.yaml ▪︎ Also after this change you can pass the secret as env variable https://github.com/flyteorg/flyteidl/pull/312 . Latest flytectl has this option ◦ part 2: Local secret : zaloando keyring is used more as a token cache and not for secret storage and hence if you don’t have GNOME keyring installed then it would just bypass the cache layer and always fetch the token by reauthenticating with the client_secret . Client_secret is stored in your filesystem eg in the functional test example its stored in /home/runner/secret_location or ENV variable cc : @Shahwar Saleem
s
Thank you for your responses.
I particularly used this method to install flytectl binary:
curl -s https://raw.githubusercontent.com/lyft/flytectl/master/install.sh |bash
For problem 1: When we install flytectl binary as mentioned, and a flyte dev deployment is running inside our VPN on AWS EKS. Flytectl is not able to resolve the IP addresses to flyte dev deployment inside the VPN. Flytectl does not use MacOS’s system name resolver, so they it not send the DNS request for flyte deployment url through our VPN
p
s
Problem 2: The suggestion of:
You can use client_secret method of auth to not use the default method which uses browser based authentication
While this suggestion can be useful for a machine-to-machine communication such as a CI/CD workflow to authenticate itself. Having generated a client secret for every user of our flyte deployment and distributing those secrets to each client is not recommended according to our company. We were hoping to see if these issues with Pkce could be resolved. Instead of opening a link in browser, is it easier to make a change that spits out a URL for the user to open however they please? Curious, How is flytectl auth tested within a container?
Could you try this one aswell curl -s https://raw.githubusercontent.com/flyteorg/flytectl/master/install.sh |bash
I think tried this one already like I said above, did you mean something else? 😛
p
This one is not using ctl.flyte.org and instead using githubusercontent
Ok i see you have tried both
👍 1
s
I actually tried this one if you saw my message above. The one you said.
p
Other option you tried to compile it locally should unblock you for some time . Is that correct ?
Not sure why this is specific to flytectl installation though since this is hosted on github repository . Are you able to install any other binaries from github repo
s
For sometime yes, but soon enough more than our team will start using flytectl and we would like to have a smooth experience for the users of workflows.
a
BTW, just to chime in (I work with Shahwar), we confirmed this like so:
Copy code
GODEBUG=netdns=2 flytectl get projects
...
go package net: built with netgo build tag; using Go's DNS resolver
...
I believe the binary needs to be compiled on a mac to allow it to use the system’s dns resolver.
Lots of discussion of this here: https://github.com/golang/go/issues/12524
p
Thanks @Andrew Achkar that is helpful. So this should be affecting all go binaries on mac IMO.
👍 1
a
Yes, though with cgo enabled and building on a mac, I believe the desired behaviour can be achieved, regardless of that open issue.
p
I think there was some investigation around enabling cgo by @Yuvraj. Adding him to the chain. This was the issue https://github.com/flyteorg/flyte/issues/1082
k
@Andrew Achkar you can build from Source right
a
On the point of not being able to open a browser (if eg. we are running flytectl inside a devcontainer), would it be a good compromise if we contributed a fallback whereby if the browser fails to open, the url it was trying to open is instead printed to the console?
p
@Andrew Achkar doesn’t the client_secret solution work for you the one i described here
• Problem 2
◦ part 1: Browser issue : You can use client_secret method of auth to not use the default method which uses browser based authentication .We use the following way in our CI by reusing the flypropeller secret https://github.com/flyteorg/flytetools/blob/master/functional-tests/config.yaml
▪︎ Also after this change you can pass the secret as env variable https://github.com/flyteorg/flyteidl/pull/312 . Latest flytectl has this option
◦ part 2: Local secret : zaloando keyring is used more as a token cache and not for secret storage and hence if you don’t have GNOME keyring installed then it would just bypass the cache layer and always fetch the token by reauthenticating with the client_secret . Client_secret is stored in your filesystem eg in the functional test example its stored in /home/runner/secret_location or ENV variable
a
Right, but this isn’t for a machine or CI use case, it is for developers/data scientists interacting with the remote. We’d rather they use auth flow over client secrets in that case.
👍 1
p
Have created a PR to log the url and also added a config to skip browser opening https://github.com/flyteorg/flyteidl/pull/313. Do you mind opening an issue so that we can link this PR to it. Also regarding the caching there can separate implementation of token cache which doesn’t use GNOME keyring. (this would avoid having the user to open browser each time) Following is the interface https://github.com/flyteorg/flyteidl/blob/master/clients/go/admin/pkce/token_cache.go for which we can add an implementation in flytectl Existing zalando keyring implementation which caters to most of the popular os’es is already there in flytectl https://github.com/flyteorg/flytectl/blob/master/pkg/pkce/token_cache_keyring.go Would love a contribution on cache implementation . @Andrew Achkar @Shahwar Saleem
❤️ 1
s
Thanks @Prafulla Mahindrakar for creating this PR. I will create a ticket shortly for this one.
👍 1
a
FWIW, the default timeout of 15 seconds was also a stumbling block for us. It look longer than 15s to get our passwords in, accept MFA prompt, etc. I would suggest to increase this default to something more generous like 2 minutes? https://github.com/flyteorg/flyteidl/blob/dbeb016d47ad38fed2430a295871fcc7191b46a9/clients/go/admin/config.go#L89
In that case the browser flow kept going but flytectl had exited before I completed the login. On future flytectl execution the browser had it cached so it went quickly and it wasn’t an issue.
p
Done @Andrew Achkar. Made the change as part of the same PR .
🎉 2
a
Thanks @Prafulla Mahindrakar!
1
p
cc ; @Haytham Abuelfutuh @katrina
Long term proposal is to use this flow https://www.oauth.com/oauth2-servers/device-flow/ @Haytham Abuelfutuh @katrina, @Shahwar Saleem and team is checking if we can still include this option of printing the url
s
Yes, if the URL can be printed onto the console in the above PR. That would unblock us from our investigations and the flyte team can work on the long term solution.
169 Views