happy-bird-19790
01/06/2022, 1:04 AMkubectl -n flyte get services flyteadmin
to find it. Is this the correct way to do it?
2. I am having trouble configuring authentication with google cloud.
Using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#deployment-cluster-config-auth-setup I did the following
+ Setup my google cloud OAuth2 Client Credential
+ Ran kubectl edit secret -n flyte flyte-admin-secrets
and added the client secret
+ Ran kubectl edit configmap -n flyte flyte-admin-config
updated the config according to the docs
+ Restarted flyteadmin with kubectl rollout restart deployment/flyteadmin -n flyte
I didn't get everything wrong because when I visited the flyte console it redirected me to google to login before going to the dashboard.
However when I tried to run a workflow the new execution just hung with status unknown.
I also was unable to connect with flytectl no matter what I tried.
I'm not sure what I'm doing wrong here. Any help is much appreciated.freezing-airport-6809
freezing-airport-6809
I don’t think I am using the correct endpoint for flytectl. I thought it should be the subdomain I access the console through but that didn’t work.
After trying a few things I was able to get it working by pointing flytectl directly to the flyteadmin service’s load balancer on port 81.
I usedIdeally you should use flyteadmin and flyteconsole behind the same domain - If flyte console is working and it is autodiscovering flyteadmin, it is because they are running on the same domain. Are you using Ingress? If so on your domain can you tryto find it. Is this the correct way to do it?kubectl -n flyte get services flyteadmin
<https://your-doman/api/v1/projects>
If this works, then your admin service port 80 is correctly configured. But, I will also let @icy-agent-73298 chip in here - on your browserfreezing-airport-6809
2. I am having trouble configuring authentication with google cloud.
Using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#deployment-cluster-config-auth-setup I did the following
+ Setup my google cloud OAuth2 Client Credential
+ Ranand added the client secretkubectl edit secret -n flyte flyte-admin-secrets
+ Ranupdated the config according to the docskubectl edit configmap -n flyte flyte-admin-config
+ Restarted flyteadmin withkubectl rollout restart deployment/flyteadmin -n flyte
I didn’t get everything wrong because when I visited the flyte console it redirected me to google to login before going to the dashboard.
However when I tried to run a workflow the new execution just hung with status unknown.
I also was unable to connect with flytectl no matter what I tried.Seems like you have not added the client secret for flytepropeller, which is the actual engine that progresses the flyte workflows and talks with FlyteAdmin. if you can
kubectl get pods -n flyte | grep flytepropeller
and then kubectl logs <pod>
freezing-airport-6809
happy-bird-19790
01/06/2022, 4:01 AMkubectl -n flyte get ingress
I see two results one named flyte-core
and the other flyte-core-grpc
both have the same host, port, and address
When I visit https://your-doman/api/v1/projects I get a json response with my projects listed.
In .flyte/config.yaml
I've update the endpoint line to endpoint: dns:///your-domain:80
from the working endpoint: dns:///aws-load-balancer:81
I'm getting the following result with flytectl
flytectl get projects
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-01-05T22:49:12-05:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-01-05T22:49:12-05:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-01-05T22:49:12-05:00"}
{"json":{"src":"client.go:180"},"level":"error","msg":"failed to initialize token source provider. Err: rpc error: code = Unavailable desc = connection closed","ts":"2022-01-05T22:49:12-05:00"}
{"json":{"src":"client.go:185"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-01-05T22:49:12-05:00"}
{"json":{"src":"client.go:58"},"level":"info","msg":"Initialized Admin client","ts":"2022-01-05T22:49:12-05:00"}
Error: rpc error: code = Unavailable desc = connection closed
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unavailable desc = connection closed","ts":"2022-01-05T22:49:13-05:00"}
happy-bird-19790
01/06/2022, 4:13 AM{"json":{"exec_id":"bsqhtfi4mr","ns":"flytesnacks-development","res_ver":"69106","routine":"worker-11","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"bsqhtfi4mr\" \u003e phase:RUNNING occurred_at:\u003cseconds:1641425632 nanos:476363641 \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]","ts":"2022-01-05T23:33:52Z"}
{"json":{"exec_id":"bsqhtfi4mr","ns":"flytesnacks-development","res_ver":"69106","routine":"worker-11","src":"executor.go:342","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]]","ts":"2022-01-05T23:33:52Z"}
{"json":{"exec_id":"bsqhtfi4mr","ns":"flytesnacks-development","res_ver":"69106","routine":"worker-11","src":"handler.go:134","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]. Is nill [%!v(MISSING)]","ts":"2022-01-05T23:33:52Z"}
E0105 23:33:52.487513 1 workers.go:102] error syncing 'flytesnacks-development/bsqhtfi4mr': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
I'm finding the docs for auth a little confusing. Do I need to follow both the OpenID Connect and OAuth2 Authorization Server sections? I was only interested in configuring authentication through google if possible, not authorization.freezing-airport-6809
freezing-airport-6809
icy-agent-73298
01/06/2022, 4:53 AMfreezing-airport-6809
icy-agent-73298
01/06/2022, 4:58 AMdns:///aws-load-balancer:81
so the difference i see now if you use the domain is that it has ssl termination and hence i am suspecting that insecure is left true in your config.yamlhappy-bird-19790
01/06/2022, 5:20 AMadmin:
endpoint: dns:///aws-load-balancer:81 # works
# endpoint: dns:///your-domain # doesn't work
authType: Pkce
insecure: true
logger:
show-source: true
level: 9
storage:
type: stow
stow:
kind: s3
config:
auth_type: iam
region: us-west-2
container: flyte-prod-service-flyte
icy-agent-73298
01/06/2022, 5:21 AMinsecure: true
to
insecure: false
icy-agent-73298
01/06/2022, 5:21 AMendpoint: dns:///your-domain
happy-bird-19790
01/06/2022, 5:22 AM{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-01-06T00:22:02-05:00"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-01-06T00:22:02-05:00"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-01-06T00:22:02-05:00"}
{"json":{"src":"client.go:180"},"level":"error","msg":"failed to initialize token source provider. Err: rpc error: code = PermissionDenied desc = Forbidden: HTTP status code 403; transport: missing content-type field","ts":"2022-01-06T00:22:02-05:00"}
{"json":{"src":"client.go:185"},"level":"warning","msg":"Starting an unauthenticated client because: can't create authenticated channel without a TokenSourceProvider","ts":"2022-01-06T00:22:02-05:00"}
{"json":{"src":"client.go:58"},"level":"info","msg":"Initialized Admin client","ts":"2022-01-06T00:22:02-05:00"}
Error: rpc error: code = PermissionDenied desc = Forbidden: HTTP status code 403; transport: missing content-type field
{"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = PermissionDenied desc = Forbidden: HTTP status code 403; transport: missing content-type field","ts":"2022-01-06T00:22:02-05:00"}
icy-agent-73298
01/06/2022, 5:25 AMhappy-bird-19790
01/06/2022, 5:26 AMicy-agent-73298
01/06/2022, 5:26 AMhappy-bird-19790
01/06/2022, 5:27 AMicy-agent-73298
01/06/2022, 6:03 AMhappy-bird-19790
01/06/2022, 6:05 AMname: flyte-prod
org_name: abovedata
providers:
aws:
region: <region>
account_id: <account_id>
modules:
- type: base
- type: external-ssl-cert
domain: "<http://flyte.abovedata.io|flyte.abovedata.io>"
private_key_file: "./cert/privkey.pem"
certificate_body_file: "./cert/cert.pem"
certificate_chain_file: "./cert/chain.pem"
- type: k8s-cluster
max_nodes: 15
- type: k8s-base
happy-bird-19790
01/06/2022, 6:06 AMload_balancer_raw_dns
that the opt apply command outputhappy-bird-19790
01/06/2022, 6:08 AMicy-agent-73298
01/06/2022, 6:08 AMthankful-minister-83577
k -n flyte get ingress
, then copy that elb address that shows up, and search for it in the aws console, the load balancer that comes up is of type ‘network’ right?thankful-minister-83577
happy-bird-19790
01/06/2022, 6:14 AMthankful-minister-83577
thankful-minister-83577
k -n ingress-nginx logs service/ingress-nginx-controller controller
btwicy-agent-73298
01/06/2022, 6:16 AMicy-agent-73298
01/06/2022, 6:17 AMhappy-bird-19790
01/06/2022, 6:22 AMicy-agent-73298
01/06/2022, 6:25 AMhappy-bird-19790
01/06/2022, 6:29 AMicy-agent-73298
01/06/2022, 6:31 AMicy-agent-73298
01/06/2022, 6:33 AMhappy-bird-19790
01/06/2022, 6:33 AMhappy-bird-19790
01/06/2022, 6:35 AM127.0.0.1 - - [06/Jan/2022:06:00:15 +0000] "GET /me HTTP/2.0" 501 131 "<https://flyte.abovedata.io/console/>" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.9 Safari/537.36" 2906 0.002 [flyte-flyteadmin-80] [] 10.0.137.42:8088 131 0.000 501 6a08211986bbd36bdd22fa2f38fb9ee3
127.0.0.1 - - [06/Jan/2022:06:01:29 +0000] "H\x00\x00\x00tj\xA8\x9E#D\x98+\xCA\xF0\xA7\xBBl\xC5\x19\xD7\x8D\xB6\x18\xEDJ\x1En\xC1\xF9xu[l\xF0E\x1D-j\xEC\xD4xL\xC9r\xC9\x15\x10u\xE0%\x86Rtg\x05fv\x86]%\xCC\x80\x0C\xE8\xCF\xAE\x00\xB5\xC0f\xC8\x8DD\xC5\x09\xF4" 400 150 "-" "-" 0 0.032 [] [] - - - - 6ce31975ecfa63d344918a94299911f7
2022/01/06 06:01:50 [crit] 130#130: *270355 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443
127.0.0.1 - - [06/Jan/2022:06:30:01 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 10aab187ee0fc030cfffb9f683a1f124
127.0.0.1 - - [06/Jan/2022:06:30:03 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 10a592ec76400045899b6582b4a491cc
127.0.0.1 - - [06/Jan/2022:06:30:05 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - c3c1d409665b23488d499744c09866f1
1
icy-agent-73298
01/06/2022, 6:37 AMicy-agent-73298
01/06/2022, 6:39 AMhappy-bird-19790
01/06/2022, 6:42 AMtls 2.0 2022-01-06T06:33:06 net/opta-flyte-prod-lb/5181b1de81b2e8a5 5128fa559e0e0e55 172.70.175.129:18738 10.0.10.27:443 194 96 3153 280 - arn:aws:acm:us-west-2:518673686532:certificate/a96b5b0f-c46f-4d68-ad84-7e6ccb064b03 - TLS_AES_128_GCM_SHA256 tlsv13 - <http://flyte.abovedata.io|flyte.abovedata.io> h2 - "h2","http/1.1"
tls 2.0 2022-01-06T06:33:03 net/opta-flyte-prod-lb/5181b1de81b2e8a5 5128fa559e0e0e55 172.70.175.129:13922 10.0.10.27:443 182 91 3152 280 - arn:aws:acm:us-west-2:518673686532:certificate/a96b5b0f-c46f-4d68-ad84-7e6ccb064b03 - TLS_AES_128_GCM_SHA256 tlsv13 - <http://flyte.abovedata.io|flyte.abovedata.io> h2 - "h2","http/1.1"
tls 2.0 2022-01-06T06:33:05 net/opta-flyte-prod-lb/5181b1de81b2e8a5 5128fa559e0e0e55 172.70.175.129:16288 10.0.10.27:443 210 104 3152 280 - arn:aws:acm:us-west-2:518673686532:certificate/a96b5b0f-c46f-4d68-ad84-7e6ccb064b03 - TLS_AES_128_GCM_SHA256 tlsv13 - <http://flyte.abovedata.io|flyte.abovedata.io> h2 - "h2","http/1.1"
tls 2.0 2022-01-06T06:34:40 net/opta-flyte-prod-lb/5181b1de81b2e8a5 5128fa559e0e0e55 172.70.175.9:38220 10.0.10.27:443 183 91 3116 280 - arn:aws:acm:us-west-2:518673686532:certificate/a96b5b0f-c46f-4d68-ad84-7e6ccb064b03 - TLS_AES_128_GCM_SHA256 tlsv13 - <http://flyte.abovedata.io|flyte.abovedata.io> h2 - "h2","http/1.1"
tls 2.0 2022-01-06T06:34:35 net/opta-flyte-prod-lb/5181b1de81b2e8a5 5128fa559e0e0e55 172.70.143.75:61152 10.0.10.27:443 379 190 685 280 - arn:aws:acm:us-west-2:518673686532:certificate/a96b5b0f-c46f-4d68-ad84-7e6ccb064b03 - TLS_AES_128_GCM_SHA256 tlsv13 - <http://flyte.abovedata.io|flyte.abovedata.io> h2 - "h2","http/1.1"
[jp:~/code/flyte/opta/aws/logs] master(+34/-11)* + cat 2022-01-06-04-19-55-50D57302AA03F02D.txt
5bc8cf1d4425a49bdda561d1cb646f612e01e61624891a83a121d15bb937416b flyte-prod-service-flyte [06/Jan/2022:03:34:55 +0000] 44.237.129.175 - VRQG12Z8D8MV0V1E REST.HEAD.BUCKET - "HEAD /flyte-prod-service-flyte HTTP/1.1" 403 AccessDenied 243 - 12 - "-" "aws-sdk-go/1.37.31 (go1.17.1; linux; amd64)" - xERUwvdicSIM15Eq1k50tRWiDWYwbT3nijd6syk0QJhjlh0sNF/OIIIrd8eITKb4uuPQB4SeKXw= - ECDHE-RSA-AES128-GCM-SHA256 - <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com> TLSv1.2 -
happy-bird-19790
01/06/2022, 6:45 AM127.0.0.1 - - [06/Jan/2022:06:38:56 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 6a24fd39725c087c20644b825d2cabdc
127.0.0.1 - - [06/Jan/2022:06:38:58 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 4e606fca6e256b06c1749869de0f8b01
127.0.0.1 - - [06/Jan/2022:06:39:01 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 2b608725dcf37090bef2d3f40630fe1d
127.0.0.1 - - [06/Jan/2022:06:42:50 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 4873377819415ca0434fd819f237f38e
127.0.0.1 - - [06/Jan/2022:06:42:52 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 838df2b70e932f9513a5440e13c907ef
icy-agent-73298
01/06/2022, 6:47 AMicy-agent-73298
01/06/2022, 6:48 AMhappy-bird-19790
01/06/2022, 7:00 AMicy-agent-73298
01/06/2022, 7:00 AMh2 2021-01-01T01:11:19.448563Z app/k8s-flyte-flytesys-f5f79a76fb/0f5c7efe7163f712 50.46.126.122:57853 192.168.156.35:31448 0.000 0.002 0.000 200 200 229 5386 "GET <https://demo.nuclyde.io:443/console/projects/flytetester/domains/development/executions/ahf8lvf2ky> HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:590375264460:targetgroup/k8s-flyte-flytecon-f4d140aa81/4e985ce8467816a6 "Root=1-5fee76b7-093e67ed0f08306856fd1714" "<http://demo.nuclyde.io|demo.nuclyde.io>" "arn:aws:acm:us-east-2:590375264460:certificate/e2f04275-2dff-4118-a493-ed3ec8f41605" 2 2021-01-01T01:11:19.445000Z "forward" "-" "-" "192.168.156.35:31448" "200" "-" "-"
icy-agent-73298
01/06/2022, 7:02 AMicy-agent-73298
01/06/2022, 7:03 AMhappy-bird-19790
01/06/2022, 7:05 AMhappy-bird-19790
01/06/2022, 7:05 AMicy-agent-73298
01/06/2022, 7:06 AMthankful-minister-83577
most-sunset-30029
01/06/2022, 5:41 PMhappy-bird-19790
01/06/2022, 9:21 PM> opta version
v0.23.0
> terraform -version
Terraform v1.0.11
on linux_amd64
> git rev-parse HEAD
8c43b3a564f2637b09c5a6a34f1a4bdc9545e68b
> git diff
diff --git a/opta/aws/env.yaml b/opta/aws/env.yaml
index 244f4494..3cd5e235 100644
--- a/opta/aws/env.yaml
+++ b/opta/aws/env.yaml
@@ -1,14 +1,16 @@
-name: <env_name>
-org_name: <your_company>
+name: flyte-prod
+org_name: abovedata
providers:
aws:
- region: <region>
- account_id: <account_id>
+ region: us-west-2
+ account_id: 518673686532
modules:
- type: base
- - type: dns
- domain: <domain>
- delegated: false # set to true once ready <https://docs.opta.dev/miscellaneous/ingress/>
+ - type: external-ssl-cert
+ domain: "<http://flyte.abovedata.io|flyte.abovedata.io>"
+ private_key_file: "./cert/live/abovedata.io/privkey.pem"
+ certificate_body_file: "./cert/live/abovedata.io/cert.pem"
+ certificate_chain_file: "./cert/live/abovedata.io/chain.pem"
- type: k8s-cluster
max_nodes: 15
- type: k8s-base
diff --git a/opta/aws/flyte.yaml b/opta/aws/flyte.yaml
index 69aa32fd..41fd99a4 100644
--- a/opta/aws/flyte.yaml
+++ b/opta/aws/flyte.yaml
@@ -2,8 +2,8 @@ environments:
- name: default
path: "./env.yaml" # NOTE: relative path to environment
variables:
- region: <region>
- account_id: <account_id>
+ region: us-west-2
+ account_id: 518673686532
name: service-flyte
modules:
- name: postgres
I installed the env with opta apply -c env.yaml --auto-approve --detailed-plan
it completed in ~25mins
I updated my cname flyte.abovedata.io to point to opta-flyte-prod-lb-500005f6805177fc.elb.us-west-2.amazonaws.com
I then installed flyte with opta apply -c flyte.yaml --auto-approve --detailed-plan
it completed in ~9mins
This is the flytectl config I'm using
> cat ~/.flyte/config.yaml
admin:
# works
endpoint: dns:///a0c0be52128614dd6a2d9f1bc17a3a15-637195292.us-west-2.elb.amazonaws.com:81
insecure: true
# # doesn't work
# endpoint: dns:///flyte.abovedata.io
# insecure: false
authType: Pkce
logger:
show-source: true
level: 9
storage:
type: stow
stow:
kind: s3
config:
auth_type: iam
region: us-west-2
container: flyte-prod-service-flyte
happy-bird-19790
01/06/2022, 9:22 PMmost-sunset-30029
01/06/2022, 9:22 PMhappy-bird-19790
01/06/2022, 9:23 PMmost-sunset-30029
01/06/2022, 9:23 PMthankful-minister-83577
thankful-minister-83577
most-sunset-30029
01/07/2022, 12:53 AMmost-sunset-30029
01/07/2022, 12:54 AMmost-sunset-30029
01/07/2022, 12:54 AMmost-sunset-30029
01/07/2022, 12:55 AMmost-sunset-30029
01/07/2022, 12:55 AMmost-sunset-30029
01/07/2022, 1:00 AMhappy-bird-19790
01/07/2022, 4:02 PMfreezing-airport-6809
happy-bird-19790
01/07/2022, 4:25 PMmost-sunset-30029
01/07/2022, 4:27 PMthankful-minister-83577
happy-bird-19790
01/07/2022, 4:57 PMmost-sunset-30029
01/07/2022, 5:00 PMhigh-park-82026
delegated: false
2. Run opta apply -c env.yaml --auto-approve
3. Run opta output -c env.yaml
and note the Name servers
4. Go to cloudflare -> DNS settings and add records for these nameservers… if it’s the root domain then use @
as the name of the record… you are going to have to add 4 records corresponding to the 4 NSs returned from opta output
5. Note that when you add NS
records, Cloudflare will automatically disable Proxying traffic… I don’t think it’ll even allow you to turn it on for these records…
6. Modify env.yaml and set delegated: true
7. Run opta apply -c env.yaml --auto-approve
to finish the rest of the domain delegation steps…high-park-82026
thankful-minister-83577
happy-bird-19790
01/07/2022, 8:38 PM{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"executor.go:269","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"debug","msg":"Transitioning/Recording event for workflow state transition [Ready] -\u003e [Running]","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"admin_eventsink.go:44","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"debug","msg":"AdminEventSink received a new event execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"x1smkw4dvh\" \u003e phase:RUNNING occurred_at:\u003cseconds:1641587535 nanos:884137893 \u003e ","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"x1smkw4dvh\" \u003e phase:RUNNING occurred_at:\u003cseconds:1641587535 nanos:884137893 \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"executor.go:342","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"executor.go:370","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"info","msg":"Handling Workflow [x1smkw4dvh] Done","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","res_ver":"118138","routine":"worker-6","src":"handler.go:134","wf":"flytesnacks:development:flyte.workflows.example.my_wf"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]. Is nill [%!v(MISSING)]","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","routine":"worker-6","src":"passthrough.go:80"},"level":"debug","msg":"Observed FlyteWorkflow Update (maybe finalizer)","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","routine":"worker-6","src":"passthrough.go:100"},"level":"debug","msg":"Updated workflow.","ts":"2022-01-07T20:32:15Z"}
{"json":{"exec_id":"x1smkw4dvh","ns":"flytesnacks-development","routine":"worker-6","src":"handler.go:284"},"level":"info","msg":"Completed processing workflow.","ts":"2022-01-07T20:32:15Z"}
E0107 20:32:15.909422 1 workers.go:102] error syncing 'flytesnacks-development/x1smkw4dvh': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
happy-bird-19790
01/07/2022, 9:12 PMkubectl edit configmap -n flyte flyte-propeller-config
and set the clientId there. But after doing that I still have the errors after restarting flytepropellerhigh-park-82026
admin
section looks like now?
And what flyte-admin-config auth
section looks like?high-park-82026
high-park-82026
thankful-minister-83577
kf get secret flyte-secret-auth -o jsonpath="{.data.client_secret}" | base64 --decode
foobar%
happy-bird-19790
01/07/2022, 9:53 PMhigh-park-82026
/etc/hosts
was messed up and it was missing 127.0.0.1 localhost
entry causing flytectl to fail in weird ways… we should experiment with using 127.0.0.1 directly as the callback url (maybe) or at the very least document the error @happy-bird-19790 posted and the solution…high-park-82026
most-sunset-30029
01/07/2022, 10:17 PMhappy-bird-19790
01/07/2022, 10:20 PMhappy-bird-19790
01/07/2022, 10:20 PMhigh-park-82026
happy-bird-19790
01/08/2022, 12:47 AMexternal-ssl-cert
option in env.yaml
So just a recap of my issues.
Issue 1
When I setup the cname in my cloudflare instance I didn't realize by default that cloudflare would proxy the requests and by default would filter out grpc. The console worked, but flytectl didn't.
Solution:
You have two options either enable grpc in the network tab or turn off the proxy both worked for me.
Issue 2
Following the auth guide to enable google oidc didn't fully work. I could login via google but running a workflow would hang.
Solution:
The docs left out that I needed to restart the flytepropeller deployment. (I went down a lot of rabbit holes here as I thought I needed to continue with the OAuth2 configuration to make it all work)
Issue 3
When I tried to run flytectl get projects
with auth enabled my browser just opened with a long localhost url. What was happening was flytectl failed to start its server but was still opening the browser.
Solution:
I made a correction to my /etc/hosts file so flytectl could start it's serverhappy-bird-19790
01/08/2022, 12:47 AM