Channels
#ask-the-community
Title
# ask-the-community
c
Chris Grass
10/05/2023, 9:59 PMhello hello - i am trying to setup workload identity in aks. i followed this guide with a few adjustements and i got the flyte pod deployed and talking to azure storage account; it creates the container, flyte and metadata blob, etc. i setup a kubernets ServiceAccount (
workload-identity-saworkload-identity-development-sapyflyte --verbose run --service-account workload-identity-development-sa --project flyte-az --domain development --remote ./workflows/simple-workflow.py simple_workflow Copy code
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to create workflow in propeller flyteworkflows.flyte.lyft.com is forbidden: User "system:serviceaccount:flyte-az:workload-identity-sa" cannot create resource "flyteworkflows" in API group "flyte.lyft.com" in the namespace "flyte-az-development": Azure does not have opinion for this user."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8089 {grpc_message:"failed to create workflow in propeller flyteworkflows.flyte.lyft.com is forbidden: User \"system:serviceaccount:flyte-az:workload-identity-sa\" cannot create resource \"flyteworkflows\" in API group \"flyte.lyft.com\" in the namespace \"flyte-az-development\":
Azure does not have opinion for this user.", grpc_status:13, created_time:"2023-10-05T15:52:41.780222-06:00"}"note that the SA referenced is the one associated with the primary flyte deployment, not the one i setup to use with tasks
i attempted to associate the
taskPodTemplate Copy code
apiVersion: v1
kind: PodTemplate
metadata:
name: service-account-template
namespace: flyte-az-development
template:
metadata:
labels:
azure.workload.identity/use: "true"
spec:
containers:
- name: default
image: {private-acr}
serviceAccountName: workload-identity-development-sai am either missing a configuration or misunderstanding it entirely (not mutually exclusive)
any ideas?
i also tagged my tasks explicitly with the
PodTemplate Copy code
@task(pod_template_name="service-account-template")
def dataframe_to_csv(df: pd.DataFrame) -> str:
csv_buffer = StringIO()
df.to_csv(csv_buffer)
csv_buffer.flush()
return csv_buffer.getvalue()d
David Espejo (he/him)
10/05/2023, 10:24 PMHi @Chris Grass !
So, are the Task pods using the Service Account you created?
Are you using
flyte-binaryflyte-corec
Chris Grass
10/05/2023, 10:25 PMhi David - the task pods appear to fail before being created. i don't see one if i run
kubectl get pods -n flyte-az-developmenti am using
flyte-binarythis is the full verbose output:
Copy code
pyflyte --verbose run --service-account workload-identity-development-sa --project flyte-az --domain development --remote ./workflows/simple-workflow.py simple_workflow
Verbose mode on
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:290 in _with_call │
│ │
│ ❱ 290 │ │ return call.result(), call │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:379 in result │
│ │
│ ❱ 379 │ │ raise self │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:1043 in with_call │
│ │
│ ❱ 1043 │ │ return _end_unary_response_blocking(state, call, True, None) │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking │
│ │
│ ❱ 910 │ │ raise _InactiveRpcError(state) # pytype: disable=not-instantiable │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯d
David Espejo (he/him)
10/05/2023, 10:47 PMwhat about the flyte-binary Pod? is it using the
ServiceAccountName: workload-identity-sac
Chris Grass
10/05/2023, 10:49 PMyes - flyte-backend-flyte-binary-db6f776b5-7phcb is using
workload-identity-sain the
flyte-azd
David Espejo (he/him)
10/05/2023, 10:52 PMAnd I guess you created the SA manually and left
serviceAccount.createc
Chris Grass
10/05/2023, 10:52 PMcorrect
d
David Espejo (he/him)
10/05/2023, 10:54 PMI think the issue here is that, when that happens, no
ClusterRoleBindinglet me see what I can find in the templates
c
Chris Grass
10/05/2023, 10:56 PMahhhh - i read that the
clusterrolerbac.createClusterRoleBinding{{- if and .Values.rbac.create .Values.serviceAccount.create }}d
David Espejo (he/him)
10/05/2023, 10:57 PMif you add the
azure.workload...commonAnnotationsc
Chris Grass
10/05/2023, 10:58 PMok, i'll walk down that path and see where it leads. thanks for the quick response!
d
David Espejo (he/him)
10/05/2023, 10:59 PMcool, let us know if it works. I think there's a growing need to have better resources for Flyte deployments on Azure
c
Chris Grass
10/05/2023, 11:03 PMcool - i work with Terence, who has a PR up to add Azure AD support to stow. and i'll put together a small PR for flyte with a couple small changes to get it up and running with azure configs
for now i hacked the clusterrolebinding yaml to only look at rbac.create - that seems to create the necessary role and unblock my testing. thanks again!