https://flyte.org logo
Join Slack
Powered by
I'm following the single cluster deployment accord...
# flyte-deployment
c
I'm following the single cluster deployment according to this link. Cloud is AWS, db is RDS, and I'm using eks managed nodes (amazon AMI). My pod almost immediately enters crashloopbackoff. I have confirmed node <> database connectivity by ssh'ing into the node and querying the RDS database. One area of suspicion is the 
serviceAccount
annotation. I have set 
create: false
and I am using the EKS cluster service role arn.
Copy code
serviceAccount:
  create: false
  annotations:
  <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<id-redacted>:role/eksctl-flyte-cluster-cluster-ServiceRole"
Copy code
2023/04/25 15:57:49 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[3.789ms] [rows:0] CREATE INDEX IF NOT EXISTS "artifacts_dataset_uuid_idx" ON "artifacts" ("dataset_uuid")
{"json":{"src":"start.go:169"},"level":"panic","msg":"Failed to start Propeller, err: failed to create FlyteWorkflow CRD: <http://customresourcedefinitions.apiextensions.k8s.io|customresourcedefinitions.apiextensions.k8s.io> is forbidden: User \"system:serviceaccount:flyte:default\" cannot create resource \"customresourcedefinitions\" in API group \"<http://apiextensions.k8s.io|apiextensions.k8s.io>\" at the cluster scope","ts":"2023-04-25T15:57:54Z"}
panic: (*logrus.Entry) 0xc0008ec540

goroutine 53 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc0008ec4d0|github.com/sirupsen/logrus.(*Entry).log(0xc0008ec4d0>, 0x0, {0xc010f68d80, 0x117})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x45b
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc0008ec4d0|github.com/sirupsen/logrus.(*Entry).Log(0xc0008ec4d0>, 0x0, {0xc00121be68?, 0x1?, 0x1?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc0008ec4d0|github.com/sirupsen/logrus.(*Entry).Logf(0xc0008ec4d0>, 0x0, {0x305c298?, 0x0?}, {0xc0084d87a0?, 0x0?, 0x0?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:338 +0x85
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3e83040|github.com/sirupsen/logrus.(*Entry).Panicf(0x3e83040>?, {0x305c298?, 0x416667?}, {0xc0084d87a0?, 0x29975a0?, 0x1?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:376 +0x34
<http://github.com/flyteorg/flytestdlib/logger.Panicf({0x3e83040|github.com/flyteorg/flytestdlib/logger.Panicf({0x3e83040>?, 0xc000943080?}, {0x305c298, 0x22}, {0xc0084d87a0, 0x1, 0x1})
        /go/pkg/mod/github.com/flyteorg/flytestdlib@v1.0.16/logger/logger.go:188 +0x64
<http://github.com/flyteorg/flyte/cmd/single.glob..func4.2()|github.com/flyteorg/flyte/cmd/single.glob..func4.2()>
        /flyteorg/build/cmd/single/start.go:169 +0xbe
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:72 +0xa5
m
@Cody Scandore the 
eksctl-flyte-cluster-cluster-ServiceRole
annotation needs to be attached to a service account; so 
create: false
is tripping you up. The IRSA annotation is needed to inform your pod's AWS credentials; for eg: allowing your flyte propeller, etc, access to s3. However, the problem you're having is that the default service account you're using doesn't have permissions to create the FlyteWorkflow CRD
the ClusterRole is created though, so in theory you could provide your default service account permissions to that role. In general, however, 
default
shouldn't really be used, and rarely used for a cluster scoped set of permissions since it can lead to privilege escalation.
c
Thanks Mike, this is really helpful
m
Oh, also, it wasn't clear above. If you do continue to use the 
default
service account; you will also need to annotate it with the IRSA annotation (i.e. `eks.amazonaws.com/role-arn`).
c
I opted to use the 
create: true
as recommended and it worked beautifully. Thanks again
196 Views
Open in Slack
PreviousNext
Powered by