https://flyte.org logo
Powered by
a

Alex Bain

04/17/2022, 11:39 PM
I just updated all my Flyte components to the new release 
v0.19.4
(i.e. Flyte Console 
v0.52.1
, Flyte Admin 
v0.6.147
, etc). When I browse to Flyte Console, instead of getting a list of my projects I get:
Copy code
Failed to fetch data
invalid wire type 4 at offset 1
Retry (button)
This looks like a gRPC binary incompatibility between the latest Flyte Console and Flyte Admin. Flyte team, could you take a look?
✅ 1
I guess it could also be some type of new CORS or auth issue. I my Flyte Admin YAML config I have:
Copy code
data:
  server:
    security:
      secure: false
      useAuth: false
      allowCors: true
since we don't bother to login (for my company Flyte deployment). However, the error message really looks like a backwards-incompatible protobuf change.
k

Ketan (kumare3)

04/17/2022, 11:58 PM
Did you upgrade both console and admin?
We have not observed this - cc @Prafulla Mahindrakar
p

Prafulla Mahindrakar

04/18/2022, 4:59 AM
I am not able to repro this issue with those versions of admin and console. Can you check admin logs . Also it might point to some ingress issue
a

Alex Bain

04/18/2022, 5:06 AM
This is definitely a coding error, but it sort of looks like the issue is in Flyte Console (and not Flyte Admin) and that it is isolated to the 
/console
page. If I browse to 
/console
in Flyte Console, I get this error and the Flyte Admin logs do not show any SQL queries.
k

Ketan (kumare3)

04/18/2022, 5:08 AM
let me try it out
a

Alex Bain

04/18/2022, 5:08 AM
But if I manually change my browser to the URL for a project I have created (e.g. `. If I manually browse to the page for a project I created (e.g. 
/console/projects/avexampleworkflows/workflows?domain=dev
for me), everything works fine and I see the queries the workflows in my Flyte Admin logs.
Ah, here is something suspicious
k

Ketan (kumare3)

04/18/2022, 5:09 AM
hmm weirdly, i tried 
<https://demo.mydomain.io/console>
and that works and returns
a

Alex Bain

04/18/2022, 5:10 AM
On the 
/console
page if I click the 
i
button in the upper right-hand corner I get:
Copy code
UI Version: 0.52.0
Admin Version:
Google Analytics: Inactive
k

Ketan (kumare3)

04/18/2022, 5:10 AM
hmm
so it is unable to connect to admin?
a

Alex Bain

04/18/2022, 5:11 AM
But on 
/console/projects/avexampleworkflows/workflows?domain=dev
I get 
0.6.14
for the Admin Version. I guess indeed, some sort of issue querying to admin from /console.
p

Prafulla Mahindrakar

04/18/2022, 5:13 AM
cc : @Nastya Rusina
Copy code
flytectl version                                                          
....
...
{
  "App": "controlPlane",
  "Build": "2c7c541",
  "Version": "v0.6.147",
  "BuildTime": "2022-04-18 04:40:28.1541366 +0000 UTC m=+0.160188001"
}%
Can you check using flytectl version command. I guess UI is clipping the last digit
Can you also check your ingress logs.
eg : in my case i do see this when i browse to /console
Copy code
[2022-04-18T05:18:51.325Z] "GET /me HTTP/1.1" 501 - 0 98 1 1 "192.168.65.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36" "8eca0678-7043-4f5d-befe-8a11c2255b1a" "localhost" "10.1.3.54:8088"
[2022-04-18T05:18:51.327Z] "GET /api/v1/projects HTTP/1.1" 200 - 0 388 7 6 "192.168.65.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36" "0e1239f1-98c4-4729-a3df-2b336b946688" "localhost" "10.1.3.54:8088"
k

Ketan (kumare3)

04/18/2022, 5:20 AM
@Prafulla Mahindrakar it is clipping
admin should be 0.6.147
a

Alex Bain

04/18/2022, 5:23 AM
Copy code
FA21110289:fabrik alex.bain$ flytectl version --admin.endpoint=avflyteadmin.scratch-alexbain.dev.l5.woven-planet.tech:443

INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 

{"json":{},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unimplemented desc = unknown service flyteidl.service.AuthMetadataService","ts":"2022-04-17T22:21:50-07:00"}
{"json":{},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-04-17T22:21:50-07:00"}
{"json":{},"level":"info","msg":"Initialized Admin client","ts":"2022-04-17T22:21:50-07:00"}

 A new release of flytectl is available: 0.4.13 → v0.5.18 
To upgrade, run: brew update && brew upgrade flytectl 
<https://github.com/flyteorg/flytectl/releases/tag/v0.5.18> 

{
  "App": "flytectl",
  "Build": "2827633",
  "Version": "0.4.13",
  "BuildTime": "2022-04-17 22:21:50.113876 -0700 PDT m=+0.017312024"
}{
  "App": "controlPlane",
  "Build": "2c7c541",
  "Version": "v0.6.147",
  "BuildTime": "2022-04-17 19:24:50.337211701 +0000 UTC m=+0.028893426"
}
Pardon the extra stuff, but for 
controlPlane
there is 
"Version": "v0.6.147"
p

Prafulla Mahindrakar

04/18/2022, 5:24 AM
Also can you try 
flytectl get projects --admin.endpoint=<http://avflyteadmin.scratch-alexbain.dev|avflyteadmin.scratch-alexbain.dev>.l5.woven-planet.tech:443
So the version clipping seems to be a UI issue
a

Alex Bain

04/18/2022, 5:25 AM
Right, I did this query earlier... which lead me to think it is some kind of UI issue in 
/console
. Great minds think alike 😉
I do get back:
Copy code
-------------------- -------------------- ------------- 
| ID                 | NAME               | DESCRIPTION |
 -------------------- -------------------- ------------- 
| avexampleworkflows | avexampleworkflows | hello       |
 -------------------- -------------------- ------------- 
1 rows
It could be some kind of timing issue... i.e. the JS is trying to use the results of the query to 
/api/v1/projects
before the query actually returns
I have to stop for tonight, but thanks for debugging! Hope we can trace the issue.
p

Prafulla Mahindrakar

04/18/2022, 5:30 AM
Can you help with the ingress logs when you hit /console flytectl uses grpc whereas console uses REST endpoints to query . So somethings off with the REST endpoints and may be its routing .
k

Ketan (kumare3)

04/18/2022, 3:13 PM
Cc @Jason Porter
n

Nastya Rusina

04/18/2022, 3:45 PM
@Alex Bain which browser do you use? We solve clipping for Chrome, and it usually easy to check by hovering on the admin version, as the link at the bottom will never be clipped. We also seen that Chrome browser started to change how they work with CORS policies, so a lot of plugins for chrome to handle CORS are stopped working or working partially… However, taking in mind that direct links are working for you this shouldn’t be the case. cc: @Jason Porter
a

Alex Bain

04/18/2022, 4:51 PM
Prafulla - it can't be an egress / ingress issue from console to admin as my pages like 
/console/projects/avexampleworkflows/workflows?domain=dev
are working fine, only 
/console
is broken.
n

Nastya Rusina

04/18/2022, 5:05 PM
@Alex Bain - If you open the Dev tools for the browser, do you see any errors in project request/response? Do you see any other api calls marked with red? Ensure to refresh page after devtools is open so all requests will go through it..
p

Prafulla Mahindrakar

04/18/2022, 5:07 PM
Ok . Is it the case that the 
/api/v1/projects
api works on 
/console/projects/avexampleworkflows/workflows?domain=dev
and not on /console page ?
a

Alex Bain

04/18/2022, 5:40 PM
I am building a few versions of 
flyteconsole
to bisect the source of the issue. I have the bug in 
v0.51.3
but not in 
v0.47.3
.
I have the issue in 
v0.50.4
, but not in 
v0.48.0
. Lol it takes a while to build and deploy. I'll find the version where the issue was introduced.
I have the issue in 
v0.50.4
but not in 
v0.50.2
which means the issue must have been introduced in 
v0.50.3
. I have a build going to confirm that. So the issue must be from https://github.com/flyteorg/flyteconsole/pull/368/files or (more likely) https://github.com/flyteorg/flyteconsole/pull/366/files.
n

Nastya Rusina

04/18/2022, 7:44 PM
That PR (second -> webpack one) only changes location for assets file (tab icon and svg icons in the graph) cc: @Jason Porter
a

Alex Bain

04/18/2022, 8:41 PM
I built the commit for #366 and that has the issue. The preceding commit (for #368) works for me.
@Nastya Rusina could you and the UI team figure out the bug in #366?
BTW that another effect of the bug is that I do not get the word "Login" in the upper-right hand corner. I just see the "i" with a circle around it.
@Nastya Rusina here is a theory. You have in your Google cache assets for the previous path. Thus, even though the code is now incorrect, your assets load anyways since they are cached in your browser. Can you try completely clearing your Chrome cache of all files, cookies, etc. and see if you get the issue then?
n

Nastya Rusina

04/18/2022, 9:14 PM
Hi @Alex Bain, I’m sorry that I slow on response today. I got a massive migraine going on, and mainly out of laptop. This change was firstly introduced in v0.49.0 , and then reverted in v0.50.2 and then re-introduced in v0.50.3 . Due to which we tested it with in two different local environment + one remote. The files location is fully changed in this case, and doesn’t included anymore a hash folder, so pre-cached values shouldn’t be an issue in this case. We got an issue going on from 0.49.0 till 0.50.1 in our demo production builds - which was mainly shown up as a pre-build error. However, just in case I just tried to fully clear cache and to enter site it from home laptop - it worked without issues.
Do you have time to have a quick call now, to see what exactly is happening on your side?
The error we saw in production before version 0.50.1 was presented in console as in the image. Is it the same to what you are seeing?
a

Alex Bain

04/18/2022, 9:21 PM
Right, I saw that in 
v0.50.0
(or 
v0.50.1
- I can't remember exactly which). But then it was fixed in 
v0.50.2
. -- But then 
v0.50.3
is broken for me :(
n

Nastya Rusina

04/18/2022, 9:22 PM
I’m curious which setup do you use? do you replace flyteconsole versions with helm? let me know if you are open to the call.
Or do you build console locally?
a

Alex Bain

04/18/2022, 9:25 PM
Or do you build console locally?
^^^^ Right, I'm building locally
n

Nastya Rusina

04/18/2022, 9:26 PM
Do you use 
yarn start
or just 
yarn build
or 
yarn build:prod
?
maybe some other command?
a

Alex Bain

04/18/2022, 9:39 PM
I build and deploy the 
Dockerfile
image to our test cluster, so the commands I run are just those from https://github.com/flyteorg/flyteconsole/blob/v0.50.3/Dockerfile. Looks like it runs 
index.js
at the end.
n

Nastya Rusina

04/18/2022, 9:41 PM
do you use M1 mac to build docker image?
a

Alex Bain

04/18/2022, 9:43 PM
No, I have an Intel Mac. I did notice the Dockerfile is running with 
NODE_ENV=production
... do you have the same?
Pardon, I was on another call until a minute ago. I do have time for a call whenever.
n

Nastya Rusina

04/18/2022, 9:45 PM
Ok, let me try to build and deploy new docker image on Intel Mac
a

Alex Bain

04/18/2022, 10:06 PM
Hmm, I have inherited the Flyte Console deployment from someone else. Looking through that last PR it touches the 
env.BASE_URL
environment variable in 
index.js
. Our deployment yaml specifically sets this environment variable to 
/console
. I'm going to unset it...
^^^^ Ah, the README does says setting BASE_URL=/console is ok.
n

Nastya Rusina

04/18/2022, 10:10 PM
yes it is required for production build
Per our call it looks more like a race condition, I will try to artificially introduce some delay into my response to reproduce the issue. The ‘Failed to fetch data’ from WaitForQuery component is shown on the screen.
Per our additional investigation Alex’s hosting the API and console on 
different
domains. And during projects request from 
/console
the api is trying to read data from 
console.domain/api/v1/projects
and if he is on any other page than it properly uses 
admin.domain/api/v1/project
.
j

Jan Willem Kleinrouweler

04/19/2022, 3:55 PM
Hello @Nastya Rusina! Similarly to @Alex Bain, I also run a Flyte deployment with the API and Console on different domains. Hence, I ran into the same problem when updating Console (from 
v0.42.0
to 
v0.54.0
). I noticed that when accessing 
/console
, the 
ADMIN_API_URL
and 
ENABLE_GA
settings seem to be ignored. When accessing 
/console/projects/<project>/workflows
the settings are correct. When inspecting the HTML source code for 
/console
, the 
window.env
is not populated with the env. variables. It makes me wonder if removing the 
assets
from here, causes 
/console
to render a static asset, thus bypassing the 
serverRenderer
, which injects the script that sets 
window.env
? I hope this helps. I’m nevertheless interesting in hearing what the solution is 🙂.
n

Nastya Rusina

04/21/2022, 3:03 AM
Hi @Jan Willem Kleinrouweler, it was connected to our files location change. The fix for the problem was just merged to flyteconsole: https://github.com/flyteorg/flyteconsole/pull/412 Thanks a lot to Alex who helped as debug the issue and test the fix. Jan, I’m curious if you have same or different console and admin domains? 🤔
Fix included in flyteconsole Release - v0.54.1
j

Jan Willem Kleinrouweler

04/21/2022, 8:32 AM
Thanks @Nastya Rusina! I can confirm that the fix is working. I indeed run the API server and Console on different domains.
🎉 1
🙌 1
11 Views
Powered by