We're trying to register workflows from within a D...
# flyte-on-gcp
k
We're trying to register workflows from within a Docker container using IAP. Previously, we used port-forwarding. We're able to register workflows using IAP from a local virtual environment, but when we try from within a container, the process hangs indefinitely. We saw a similar issue in this thread, but we would like to register workflows from a user account instead of a service account. In our Google Cloud Console logs for the load balancer, we see permission denied errors and no data under
authenticationInfo
when attempting to register from the container. Contrastingly, successful registration attempts from outside of the container show
principalEmail
with my email under
authenticationInfo
. We're also unable to run the script from step 8 of the configuring IAP instructions inside the container — this hangs indefinitely as well. Does anyone have suggestions to get this working?
f
Hey Keaton, thinking about it, I think I know what happens: • Flytekit is configured with a proxy auth command which will run in a subprocess using the CommandAuthenticator • If you go for proxy auth with a user token, the
GCPIdentityAwareProxyAuthenticator
from the iap plugin will be invoked in the subprocess. It uses the
AuthorizationClient
from flytekit here which will open the webbrowser during the auth flow here. • Now comes the part that causes the hanging I think: ◦ Outside of the docker container
webbrowser.open_new_tab(
should just open the browser automatically. ◦ I think inside the docker container, this likely fails (unless it is e.g. a VSCode devcontainer where I know this still works) and prompts the user to click on the authorization link manually. ◦ Since this is run in a sub process by the CommandAuthenticator though, the user never sees this prompt, causing the hanging you see. To confirm my suspicion is correct, could you please run the
flyte-iap
command directly in the docker container? I guess it will ask you to click on the browser window manually.
If this is true, to unblock you for now, I have the following idea for a workaround: • Configure e.g.
["echo", "$FLYTE_PROXY_AUTH"]
as proxy command • Run
FLYTE_PROXY_AUTH=$(flyte-iap ...) pyflyte run ...
• This takes the “opening of the webbrowser” out of the subprocess so that the user sees the link. We then should discuss whether there is a way to enhance
CommandAuthenticator
to forward e.g. stderr to the user (for e.g. the log link or other error information) until a token is logged to stdout. Or parse all log lines of stdout and forward to the user if they are not a token … This way, the user would also know what is happening in the subprocess.
Let me know if what I wrote raised more questions pls 🙂
o
Thanks Fabio! You're right and the flyte-iap command does work when run directly. That temporary unblocker doesn't work, I think because the subprocess.run won't expand env variables without shell=True, but we can do "flyte-iap ... ", save the last output to file, and ["cat", "file"] as the proxy command for now.