The problem with this approach is that I (seemingly) can't specify the securitycontext for the sidecar containers (if I add them to the pod template, flyte will duplicate the definitions in the pod manifest and k8s will complain that there are containers with duplicate name...)

@Vladyslav Libov I think using a

default

container name in the PodTemplate will turn its spec into what Flyte uses for all containers in a Pod, not only the primary. I'll try to repro this behavior and will let you know

Thanks! Looking forward to your check!

great to hear! yeah, I forgot to mention that this works by using a Runtime Pod Template, like you just did. What's the intended structure of your Pod? How many init vs "primary" vs sidecar containers are expected? I'm not sure if the default one would cover init but we can find a way hopefully

Thanks again for looking into this! I don't explicitely define any sidecar or init containers, so those must be some default flyte settings. In the end the pod contains one "primary" container, one sidecar ("flyte-copilot-sidecar") and one init-container ("flyte-copilot-downloader"). I could give you more details, like the resulting pod manifest, or the task definition, etc.

yeah, please share your

PodTemplate

definition

This is my (k8s) `PodTemplate`:

apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    securityContext:
      runAsNonRoot: true
      seccompProfile:
        type: "RuntimeDefault"
      runAsUser: 1000
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
            - SYS_PTRACE
            - ALL

And these are defintions in the workflow-python-file: 1. The template

pod_template_data_management = Pod(
    pod_spec=V1PodSpec(
        containers=[
            V1Container(
                name="primary",
                resources=V1ResourceRequirements(
                    limits={"memory": "16Gi", "cpu": "0.8"},
                    requests={"memory": "500Mi", "cpu": "0.25"},
                ),
                image_pull_policy="Always",
            )
        ],
        image_pull_secrets=[
            # the secrets referenced here must be defined beforehand (in the flux-config repository)
            V1LocalObjectReference("imagepullsecrets")
        ],
        # node_selector = {"<http://kubernetes.io/hostname|kubernetes.io/hostname>" : "srv-k8s-test-cpu1"}
    ),
)

2. The task itself, using both the runtime (

flyte-template

) and compile time (

pod_template_data_management

) templates

download_data_from_minio = ContainerTask(
    name="DownloadData",
    image=FLYTE_IMAGE,
    command=["python", "-m", "src.download_data"],
    arguments=["--file_prefix", "{{.inputs.file_prefix}}",
               "--bucket_name", "{{.inputs.bucket_name}}",
               "--output_path", MOUNT_PATH,
                "--output_filename", "kodi_preprocessed_data",
               '-vvvv'
    ],
    output_data_dir=MOUNT_PATH,
    inputs=kwtypes(
        file_prefix = str,
        bucket_name = str
    ),
    outputs=kwtypes(
        kodi_preprocessed_data=FlyteFile
    ),
    pod_template=pod_template_data_management,
    pod_template_name="flyte-template"
)

but, regarding the Runtime PodTemplate, did you configure the

plugins.k8s.default-pod-template-name

key?

not really.... before trying the Runtime Podtemplate i tried

plugins.k8s.default-security-context

and

plugins.k8s.default-pod-security-context

(i.e. setting the security-context parameters directly there) but that didn't work (perhaps it's not correct at all). ok so I can try setting

plugins.k8s.default-pod-template-name

! (didnt do so as I thought it would be redundant to using it in the task directly). Would then

pod_template_name=...

in the task definition be unnecessary?

well I'm not sure what would be the behavior when combining multiple types of PodTemplates. To use the Runtime ones (this is, a K8s PodTemplate resource created either on the execution namespace or flyte's namespace), then you should tell Propeller to use that one and nothing else. This is achieved by setting that key to the name of your K8s PodTemplate

I'll try and get back to you

Hey Vladyslav, yes I asked around and seems like overriding the copilot containers config is not supported yet in Flyte

thanks! just to clarify: it does work for sidecarContainers, just not for initContainers. Two last questions 🙂 1. is it perhaps possible to run the initContainer as a sidecar instead? Probably not, because it has to strictly be finished before the main container starts (so that it has the necessary input data). 2. What about the other problem I mentioned? Its about the SYS_PTRACE capability. It is explicitely attached to the primary container (

securityContext.capabilities.add=['SYS_PTRACE']

, and even though I can effectively override it by coniguring the podTemplate (

securityContext.capabilities.drop=['SYS_PTRACE']

), the original add-statement remains in the manifest and makes the PodSecurity Admission Controller complain about it (even in the baseline version).

1. After checking the repo,I don't see a way to achieve that 2. Can you share the resulting Pod manifest after you drop the cap with the PodTemplate?

Hi David, this is the relevant part of the final pod manifest:

securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - SYS_PTRACE
        drop:
        - SYS_PTRACE
        - ALL

The relevant part in the template is

containers:
      - name: default
        securityContext:
            drop:
            - SYS_PTRACE
            - ALL

I also had a quick look at the source code, looks like the cap is added in flyteplugins/go/tasks/pluginmachinery/flytek8s/copilot.go as follows

var pTraceCapability = v1.Capability("SYS_PTRACE")
<.....>
c.SecurityContext.Capabilities.Add = append(c.SecurityContext.Capabilities.Add, pTraceCapability)

Doesn't seem to be configurable... Thanks!

hm do you think this is an area where you'd be able to contribute? I mean, making it configurable

yeah, would be a pleasure!

awesome! so let's start with an Issue describing the problem

Greetings David, is this understandable? https://github.com/flyteorg/flyte/issues/5462 Cheers Vladyslav

yes it is, thank you!

Hi David, quick question. My understanding is the all of the flyte-magic is happening in the flyte-binary image (we are using the flyte-binary helm chart), so whenever I want to test my chagnes to the code, all I need to do is to build a new flyte-binary image and update the reference to the image in the values.yaml accordingly. Is this correct or is there more to it? Is building an image a trivial "docker build .. "? Thanks!

hey Vladyslav sure, let's continue there. I'm waiting for some of the other maintainers to chime in as there is a chance to remove that capability altogether

yeah, just saw your comment there - thanks!

Hi @Vladyslav Libov @David Espejo (he/him) I am running flyte on fargate and got this error when running a R script - Raw Container

6/14/2024 3:13:01 PM UTC task submitted to K8s

6/14/2024 3:13:01 PM UTC POD_UNSUPPORTED_ON_FARGATE:Pod not supported on Fargate: invalid SecurityContext fields: Capabilities added: SYS_PTRACE

Will this fix my issue as well? For now i have created a flytekit wrapper around the R code and triggered subprocess for the R scripts i need. Just wondering if down the line i can switch to raw containers itself.

Hi @Gaurav Panta I guess this might solve your issue as well!

@Vladyslav Libov thank you, will wait for the release cheers!

I'll let you know!