Chris Grass
10/05/2023, 9:59 PMworkload-identity-sa
) for this purpose and associated it with an AZ id with the necessary permissions. I then setup a second kubernetes sa (workload-identity-development-sa
) to run workflows/tasks.
when i try to run pyflyte --verbose run --service-account workload-identity-development-sa --project flyte-az --domain development --remote ./workflows/simple-workflow.py simple_workflow
i get the following error:
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to create workflow in propeller flyteworkflows.flyte.lyft.com is forbidden: User "system:serviceaccount:flyte-az:workload-identity-sa" cannot create resource "flyteworkflows" in API group "flyte.lyft.com" in the namespace "flyte-az-development": Azure does not have opinion for this user."
debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8089 {grpc_message:"failed to create workflow in propeller flyteworkflows.flyte.lyft.com is forbidden: User \"system:serviceaccount:flyte-az:workload-identity-sa\" cannot create resource \"flyteworkflows\" in API group \"flyte.lyft.com\" in the namespace \"flyte-az-development\":
Azure does not have opinion for this user.", grpc_status:13, created_time:"2023-10-05T15:52:41.780222-06:00"}"
task
SA with the task pods by creating a PodTemplate
, which is referenced in my values.yaml:
apiVersion: v1
kind: PodTemplate
metadata:
name: service-account-template
namespace: flyte-az-development
template:
metadata:
labels:
azure.workload.identity/use: "true"
spec:
containers:
- name: default
image: {private-acr}
serviceAccountName: workload-identity-development-sa
PodTemplate
, hoping that would force them to adopt the referenced SA:
@task(pod_template_name="service-account-template")
def dataframe_to_csv(df: pd.DataFrame) -> str:
csv_buffer = StringIO()
df.to_csv(csv_buffer)
csv_buffer.flush()
return csv_buffer.getvalue()
David Espejo (he/him)
10/05/2023, 10:24 PMflyte-binary
or flyte-core
?Chris Grass
10/05/2023, 10:25 PMkubectl get pods -n flyte-az-development
flyte-binary
pyflyte --verbose run --service-account workload-identity-development-sa --project flyte-az --domain development --remote ./workflows/simple-workflow.py simple_workflow
Verbose mode on
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:301 in with_call │
│ │
│ ❱ 301 │ │ return self._with_call(request, │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:290 in _with_call │
│ │
│ ❱ 290 │ │ return call.result(), call │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:379 in result │
│ │
│ ❱ 379 │ │ raise self │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_interceptor.py:274 in continuation │
│ │
│ ❱ 274 │ │ │ │ response, call = self._thunk(new_method).with_call( │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:1043 in with_call │
│ │
│ ❱ 1043 │ │ return _end_unary_response_blocking(state, call, True, None) │
│ │
│ /Users/chris.grass/Library/Python/3.9/lib/python/site-packages/grpc/_channel.py:910 in _end_unary_response_blocking │
│ │
│ ❱ 910 │ │ raise _InactiveRpcError(state) # pytype: disable=not-instantiable │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
David Espejo (he/him)
10/05/2023, 10:47 PMServiceAccountName: workload-identity-sa
?Chris Grass
10/05/2023, 10:49 PMworkload-identity-sa
flyte-az
namespaceDavid Espejo (he/him)
10/05/2023, 10:52 PMserviceAccount.create
as `false`in the values file?Chris Grass
10/05/2023, 10:52 PMDavid Espejo (he/him)
10/05/2023, 10:54 PMClusterRoleBinding
is created (see https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/templates/clusterrolebinding.yaml) so this SA is kinda isolated. The error message mentions an action and API group that is not allowedChris Grass
10/05/2023, 10:56 PMclusterrole
is ignored if rbac.create
is false and thought i read the same regarding the ClusterRoleBinding
. but you are absolutely correct, that uses {{- if and .Values.rbac.create .Values.serviceAccount.create }}
David Espejo (he/him)
10/05/2023, 10:57 PMazure.workload...
annotation to commonAnnotations
in your values file, it will be added to the service account ( see https://github.com/flyteorg/flyte/blob/4ee73f583d39fb878d1c487b3e92c61e7abab329/charts/flyte-binary/templates/serviceaccount.yaml#L15C6-L15C6)Chris Grass
10/05/2023, 10:58 PMDavid Espejo (he/him)
10/05/2023, 10:59 PMChris Grass
10/05/2023, 11:03 PM