The problem with this approach is that I (seemingl...
# ask-the-community
v
The problem with this approach is that I (seemingly) can't specify the securitycontext for the sidecar containers (if I add them to the pod template, flyte will duplicate the definitions in the pod manifest and k8s will complain that there are containers with duplicate name...)
d
@Vladyslav Libov I think using a
default
container name in the PodTemplate will turn its spec into what Flyte uses for all containers in a Pod, not only the primary. I'll try to repro this behavior and will let you know
v
Thanks! Looking forward to your check!
Setting
name="default"
in the PodTemplate seems to create another container, without affecting the others...
Update: i tried creating a PodTemplate resource in k8s directly (instead of creating a PodTemplate of flytekit SDK as shown above) with name="default" and it does the trick for the main containers! The remainng problem is the init-container... Any ideas? (setting the "default" for the init-container won'T work due to name conflict in the main-container section..)
d
great to hear! yeah, I forgot to mention that this works by using a Runtime Pod Template, like you just did. What's the intended structure of your Pod? How many init vs "primary" vs sidecar containers are expected? I'm not sure if the default one would cover init but we can find a way hopefully
v
Thanks again for looking into this! I don't explicitely define any sidecar or init containers, so those must be some default flyte settings. In the end the pod contains one "primary" container, one sidecar ("flyte-copilot-sidecar") and one init-container ("flyte-copilot-downloader"). I could give you more details, like the resulting pod manifest, or the task definition, etc.
d
yeah, please share your
PodTemplate
definition
v
This is my (k8s) `PodTemplate`:
Copy code
apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    securityContext:
      runAsNonRoot: true
      seccompProfile:
        type: "RuntimeDefault"
      runAsUser: 1000
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
            - SYS_PTRACE
            - ALL
And these are defintions in the workflow-python-file: 1. The template
Copy code
pod_template_data_management = Pod(
    pod_spec=V1PodSpec(
        containers=[
            V1Container(
                name="primary",
                resources=V1ResourceRequirements(
                    limits={"memory": "16Gi", "cpu": "0.8"},
                    requests={"memory": "500Mi", "cpu": "0.25"},
                ),
                image_pull_policy="Always",
            )
        ],
        image_pull_secrets=[
            # the secrets referenced here must be defined beforehand (in the flux-config repository)
            V1LocalObjectReference("imagepullsecrets")
        ],
        # node_selector = {"<http://kubernetes.io/hostname|kubernetes.io/hostname>" : "srv-k8s-test-cpu1"}
    ),
)
2. The task itself, using both the runtime (
flyte-template
) and compile time (
pod_template_data_management
) templates
Copy code
download_data_from_minio = ContainerTask(
    name="DownloadData",
    image=FLYTE_IMAGE,
    command=["python", "-m", "src.download_data"],
    arguments=["--file_prefix", "{{.inputs.file_prefix}}",
               "--bucket_name", "{{.inputs.bucket_name}}",
               "--output_path", MOUNT_PATH,
                "--output_filename", "kodi_preprocessed_data",
               '-vvvv'
    ],
    output_data_dir=MOUNT_PATH,
    inputs=kwtypes(
        file_prefix = str,
        bucket_name = str
    ),
    outputs=kwtypes(
        kodi_preprocessed_data=FlyteFile
    ),
    pod_template=pod_template_data_management,
    pod_template_name="flyte-template"
)
d
but, regarding the Runtime PodTemplate, did you configure the
plugins.k8s.default-pod-template-name
key?
v
not really.... before trying the Runtime Podtemplate i tried
plugins.k8s.default-security-context
and
plugins.k8s.default-pod-security-context
(i.e. setting the security-context parameters directly there) but that didn't work (perhaps it's not correct at all). ok so I can try setting
plugins.k8s.default-pod-template-name
! (didnt do so as I thought it would be redundant to using it in the task directly). Would then
pod_template_name=...
in the task definition be unnecessary?
thanks again!
d
well I'm not sure what would be the behavior when combining multiple types of PodTemplates. To use the Runtime ones (this is, a K8s PodTemplate resource created either on the execution namespace or flyte's namespace), then you should tell Propeller to use that one and nothing else. This is achieved by setting that key to the name of your K8s PodTemplate
v
I'll try and get back to you
Hi David, I tried the global setting (plugins.k8s.default-pod-template-name) - it seems to be equivalent to setting the pod_template_name directly in the task definition (tried this both with and without the additional Runtime PodTemplate). Either way, the init cointainer flyte-copilot-downloader seems to be unaffected... Can it be that the initContainers are not configured by templates at all? Do you know if anyone has dealt with PodSecurityStandards with flyte? (i.e. using restricted or baseline for the corresponding namespaces)? Thanks!
d
Hey Vladyslav, yes I asked around and seems like overriding the copilot containers config is not supported yet in Flyte
v
thanks! just to clarify: it does work for sidecarContainers, just not for initContainers. Two last questions 🙂 1. is it perhaps possible to run the initContainer as a sidecar instead? Probably not, because it has to strictly be finished before the main container starts (so that it has the necessary input data). 2. What about the other problem I mentioned? Its about the SYS_PTRACE capability. It is explicitely attached to the primary container (
securityContext.capabilities.add=['SYS_PTRACE']
, and even though I can effectively override it by coniguring the podTemplate (
securityContext.capabilities.drop=['SYS_PTRACE']
), the original add-statement remains in the manifest and makes the PodSecurity Admission Controller complain about it (even in the baseline version).
d
1. After checking the repo,I don't see a way to achieve that 2. Can you share the resulting Pod manifest after you drop the cap with the PodTemplate?
v
Hi David, this is the relevant part of the final pod manifest:
Copy code
securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - SYS_PTRACE
        drop:
        - SYS_PTRACE
        - ALL
The relevant part in the template is
Copy code
containers:
      - name: default
        securityContext:
            drop:
            - SYS_PTRACE
            - ALL
I also had a quick look at the source code, looks like the cap is added in flyteplugins/go/tasks/pluginmachinery/flytek8s/copilot.go as follows
Copy code
var pTraceCapability = v1.Capability("SYS_PTRACE")
<.....>
c.SecurityContext.Capabilities.Add = append(c.SecurityContext.Capabilities.Add, pTraceCapability)
Doesn't seem to be configurable... Thanks!
PS. it's added only to the "primary". The sidecars/inits don't have it.
d
hm do you think this is an area where you'd be able to contribute? I mean, making it configurable
v
yeah, would be a pleasure!
d
awesome! so let's start with an Issue describing the problem
v
Greetings David, is this understandable? https://github.com/flyteorg/flyte/issues/5462 Cheers Vladyslav
d
yes it is, thank you!