Thread
#announcements
    Matheus Moreno

    Matheus Moreno

    5 months ago
    Hi, everyone! Quick question: is it possible to raise the memory limit on a Flyte sandbox? A member of my team is trying to test their system locally, but is being greeted with an OOMKilled error.
    k

    katrina

    5 months ago
    hi @Matheus Moreno do you mean the memory limit for an individual task or for docker itself?
    Matheus Moreno

    Matheus Moreno

    5 months ago
    for an individual task. if I try to request more than 1Gi, I get an error on the registration process
    k

    katrina

    5 months ago
    ah, for that we need to update the task limits in the flyteadmin configmap. can i ask how your teammate is starting up flytesandbox?
    Matheus Moreno

    Matheus Moreno

    5 months ago
    she's using docker directly (actually, docker-compose). can it be configured from there?
    k

    katrina

    5 months ago
    ok, so we need to modify the flyteadmin config map request limits here. also sorry, what do you mean by
    docker directly
    ?
    Matheus Moreno

    Matheus Moreno

    5 months ago
    oh, i just meant that she's not using flytectl, but actually using
    docker run ...
    to start the server
    but how can I edit this file? bypassing the entrypoint of the sandbox to first edit it?
    k

    katrina

    5 months ago
    can you share the full docker run command?
    Matheus Moreno

    Matheus Moreno

    5 months ago
    docker create --name flyte-sandbox \
        --privileged \
        --mount type=bind,src=$PWD,dst=/root \
        -p 30081:30081 -p 30082:30082 -p 30084:30084 -p 30086-30088:30086-30088 \
        <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    
    docker run flyte-sandbox            # Start sandbox
    the compose is
    version: "3.5"
    
    services:
      sandbox:
        image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
        container_name: flyte-sandbox
        privileged: true
        volumes:
          - ../..:/root
        ports:
          - "30081:30081"         # Console at <http://localhost:30081/console>
          - "30082:30082"
          - "30084:30084"
          - "30086:30086"
          - "30087:30087"
          - "30088:30088"
    (it's in a subsubdirectory)
    k

    katrina

    5 months ago
    do you have a way to modify the kubernetes resources brought up by the sandbox?
    Matheus Moreno

    Matheus Moreno

    5 months ago
    trying to do that rn
    I'm trying to add a short script before the entrypoint command to replace the value on values.yaml
    this is a VERY extreme go horse solution but it works, I think. I installed pip and yq to make the substitution (I did not want to spend hours trying to make the right sed command):
    entrypoint: >
          bash -c "
            apk add --update py-pip &&
            pip install yq &&
            yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
            tini flyte-entrypoint.sh
          "
    k

    katrina

    5 months ago
    amazing!! thanks for sharing
    Matheus Moreno

    Matheus Moreno

    5 months ago
    welp, it did not work. I changed the values on both
    /flyteorg/share/flyte/values.yaml
    and
    /flyteorg/share/flyte-core/values.yaml
    and this error still happens during registration:
    Error: rpc error: code = InvalidArgument desc = Requested MEMORY default [4Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
    i confirmed that the substitution process worked by entering the sandbox container and inspecting these files.
    .flyte.configmap.task_resource_defaults.task_resources.limits.memory
    is configured to 4Gi to both
    what could be the problem? 😕
    nevermind, fixed it! I had to specify the
    FLYTE_TEST
    variable to
    local
    , because the default is using the release values
    final compose:
    version: "3.5"
    
    services:
      sandbox:
        image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
        container_name: flyte-sandbox
        privileged: true
        volumes:
          - ../..:/root
        entrypoint: >
          bash -c "
            apk add --update py-pip &&
            pip install yq &&
            yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
            tini flyte-entrypoint.sh
          "
        environment:
          - FLYTE_TEST=local
        ports:
          - "30081:30081"         # Console at <http://localhost:30081/console>
          - "30082:30082"
          - "30084:30084"
          - "30086:30086"
          - "30087:30087"
          - "30088:30088"
    k

    katrina

    4 months ago
    thank you again for sharing!