#7034 [V2]Sandbox webhook Pull request opened by popojk Tracking issue Closes #7003 Why are the changes needed? To handle secret injection from k8s cluster secrets to flyte task pods, we implement a webhook feature in this PR. Whenever a pod create request sent to the k8s api interface, k8s will redirect the request to our webhook server. The webhook server will mutate the pod spec by adding k8s secrets to it, and the webhook will finally create pods with those secret ENVVAR. Please see bellow diagram for better understanding.

┌─────────────────────────────────────────────────────────────────────────┐
  │  Client (flytekit SDK)                                                  │
  │                                                                         │
  │  @env.task(secrets=Secret(group="my-secret", key="api-key"))            │
  │       │                                                                 │
  │       │  TaskTemplate.SecurityContext.Secrets = [{group, key}]          │
  └───────┼─────────────────────────────────────────────────────────────────┘
          │ flyte.run()
          ▼
  ┌───────────────────┐
  │   Flyte API       │  creates
  │   Server          │ ──────────────► TaskAction CRD
  └───────────────────┘                 (Spec.TaskTemplate = protobuf bytes)
                                                │
                                                │ controller watches
                                                ▼
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Executor (controller-runtime)                                          │
  │                                                                         │
  │  TaskActionReconciler.Reconcile()                                       │
  │       │                                                                 │
  │       ├─ NewTaskExecutionMetadata(ta)                                   │
  │       │    └─ extractSecurityContextFromTaskTemplate()                  │
  │       │         └─ SecurityContext.Secrets = [{group, key}]             │
  │       │                                                                 │
  │       └─ plugin.BuildResource(taskCtx)                  [pod plugin]   │
  │            ├─ flytek8s.BuildRawPod()                                    │
  │            └─ if len(SecurityContext.Secrets) > 0:                      │
  │                 pod.Labels["inject-flyte-secrets"] = "true"             │
  │                 pod.Annotations["flyte.secrets/s0"] = <encoded proto>   │
  └─────────────────────────────────┬───────────────────────────────────────┘
                                    │ client.Create(pod)
                                    ▼
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Kubernetes API Server                                                  │
  │                                                                         │
  │  MutatingWebhookConfiguration                                           │
  │    objectSelector: inject-flyte-secrets=true   ◄── only tagged pods    │
  │    clientConfig.service: flyte-pod-webhook                              │
  └─────────────────────────────────┬───────────────────────────────────────┘
                                    │ forwards pod create request
                                    ▼
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Webhook Server (same process as Executor, port 9443)                  │
  │                                                                         │
  │  PodMutator.Handle()                                                    │
  │       │                                                                 │
  │       └─ SecretsPodMutator.Mutate()                                     │
  │            ├─ UnmarshalStringMapToSecrets(pod.Annotations)              │
  │            │    └─ [{group:"my-secret", key:"api-key", ENV_VAR}]        │
  │            │                                                            │
  │            └─ K8sSecretInjector.Inject()                               │
  │                 └─ add env var to pod spec:                             │
  │                      name:  _UNION_MY-SECRET_API-KEY                   │
  │                      valueFrom.secretKeyRef:                            │
  │                        name: my-secret   (k8s Secret)                  │
  │                        key:  api-key                                    │
  └─────────────────────────────────┬───────────────────────────────────────┘
                                    │ mutated pod
                                    ▼
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Task Pod                                                               │
  │                                                                         │
  │  env:                                                                   │
  │    _UNION_MY-SECRET_API-KEY=<value from k8s Secret>                    │
  │    FLYTE_SECRETS_ENV_PREFIX=_UNION_                                     │
  └─────────────────────────────────────────────────────────────────────────┘

What changes were proposed in this pull request? • Add a webhook server in controller runtime to handle pod mutation • Add a service in helm chart in sandbox to expose webhook server • Add logic in taskExecutionMetadata to consume and handle security context in task template Follow up We should add an api endpoint in the backend for the user to add k8s secret with flyte cli i/o calling kubectl command How was this patch tested? After sandbox up and running, we set a test secret in k8s cluster first:

kubectl create secret generic webhooktest \
--from-literal=apikey=hello-from-webhook \
-n flyte

Run bellow flyte workflow which has. secret config in it's task ENV:

import asyncio
import logging
import os
from typing import List

import flyte

env = flyte.TaskEnvironment(
    name="hello_world",
    resources=flyte.Resources(cpu=1, memory="1Gi"),
    secrets=flyte.Secret(group="webhooktest", key="apikey"),
)


@env.task
async def say_hello(data: str, lt: List[int]) -> str:
    print(f"Hello, world! - {flyte.ctx().action}")
    secret_val = os.environ.get("_UNION_WEBHOOKTEST_APIKEY", "NOT_FOUND")
    print(f"[webhook test] _UNION_WEBHOOKTEST_APIKEY={secret_val}")
    return f"Hello {data} {lt} secret={secret_val}"


@env.task
async def square(i: int = 3) -> int:
    print(flyte.ctx().action)
    return i * i


@env.task
async def say_hello_nested(data: str = "default string", n: int = 3) -> str:
    print(f"Hello, nested! - {flyte.ctx().action}")
    coros = []
    for i in range(n):
        coros.append(square(i=i))

    vals = await asyncio.gather(*coros)
    return await say_hello(data=data, lt=vals)


if __name__ == "__main__":
    flyte.init_from_config()
    run = flyte.with_runcontext(
        log_level=logging.DEBUG,
        env_vars={"KEY": "V"},
        labels={"Label1": "V1"},
        annotations={"Ann": "ann"},
        overwrite_cache=True,
        interruptible=False,
    ).run(say_hello_nested, data="hello world", n=10)
    print(run.name)
    print(run.url)
    run.wait()
    # print(run.outputs())

We will see secret ENV injected by webhook in task pod: <https://private-user-images.githubusercontent.com/66259759/563882127-868828a5-12cb-472c-a44b-0d511d96801c.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzM5NTQwMTksIm5iZiI6MTc3Mzk1MzcxOSwicGF0aCI6Ii82NjI1OTc1OS81NjM4ODIxMjctODY4ODI4YTUtMTJjYi00NzJjLWE0NGItMGQ1MTFkOTY4MDFjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzE5VDIwNTUxOVomWC1Bb… flyteorg/flyte