Hi, everyone! Quick question: is it possible to raise the memory limit on a Flyte sandbox? A member ...

sparse-window-1536

04/27/2022, 6:50 PM

Hi, everyone! Quick question: is it possible to raise the memory limit on a Flyte sandbox? A member of my team is trying to test their system locally, but is being greeted with an OOMKilled error.

acceptable-policeman-57188

04/27/2022, 7:45 PM

hi @sparse-window-1536 do you mean the memory limit for an individual task or for docker itself?

sparse-window-1536

04/27/2022, 8:43 PM

for an individual task. if I try to request more than 1Gi, I get an error on the registration process

acceptable-policeman-57188

04/27/2022, 8:58 PM

ah, for that we need to update the task limits in the flyteadmin configmap. can i ask how your teammate is starting up flytesandbox?

sparse-window-1536

04/27/2022, 9:43 PM

she's using docker directly (actually, docker-compose). can it be configured from there?

acceptable-policeman-57188

04/27/2022, 9:59 PM

ok, so we need to modify the flyteadmin config map request limits here. also sorry, what do you mean by

docker directly

sparse-window-1536

04/27/2022, 10:01 PM

oh, i just meant that she's not using flytectl, but actually using

docker run ...

to start the server

sparse-window-1536

04/27/2022, 10:02 PM

but how can I edit this file? bypassing the entrypoint of the sandbox to first edit it?

acceptable-policeman-57188

04/27/2022, 10:03 PM

can you share the full docker run command?

sparse-window-1536

04/27/2022, 10:03 PM

Copy code

docker create --name flyte-sandbox \
    --privileged \
    --mount type=bind,src=$PWD,dst=/root \
    -p 30081:30081 -p 30082:30082 -p 30084:30084 -p 30086-30088:30086-30088 \
    <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>

docker run flyte-sandbox            # Start sandbox

sparse-window-1536

04/27/2022, 10:55 PM

the compose is

Copy code

version: "3.5"

services:
  sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    container_name: flyte-sandbox
    privileged: true
    volumes:
      - ../..:/root
    ports:
      - "30081:30081"         # Console at <http://localhost:30081/console>
      - "30082:30082"
      - "30084:30084"
      - "30086:30086"
      - "30087:30087"
      - "30088:30088"

sparse-window-1536

04/27/2022, 10:55 PM

(it's in a subsubdirectory)

acceptable-policeman-57188

04/27/2022, 11:03 PM

do you have a way to modify the kubernetes resources brought up by the sandbox?

sparse-window-1536

04/27/2022, 11:07 PM

trying to do that rn

sparse-window-1536

04/27/2022, 11:08 PM

I'm trying to add a short script before the entrypoint command to replace the value on values.yaml

🙏 1

sparse-window-1536

04/28/2022, 12:10 AM

this is a VERY extreme go horse solution but it works, I think. I installed pip and yq to make the substitution (I did not want to spend hours trying to make the right sed command):

Copy code

entrypoint: >
      bash -c "
        apk add --update py-pip &&
        pip install yq &&
        yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
        tini flyte-entrypoint.sh
      "

acceptable-policeman-57188

04/28/2022, 12:12 AM

amazing!! thanks for sharing

sparse-window-1536

04/28/2022, 12:35 AM

welp, it did not work. I changed the values on both

/flyteorg/share/flyte/values.yaml

and

/flyteorg/share/flyte-core/values.yaml

and this error still happens during registration:

Copy code

Error: rpc error: code = InvalidArgument desc = Requested MEMORY default [4Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration

sparse-window-1536

04/28/2022, 12:36 AM

i confirmed that the substitution process worked by entering the sandbox container and inspecting these files.

.flyte.configmap.task_resource_defaults.task_resources.limits.memory

is configured to 4Gi to both

sparse-window-1536

04/28/2022, 12:36 AM

what could be the problem? 😕

sparse-window-1536

04/28/2022, 1:04 AM

nevermind, fixed it! I had to specify the

FLYTE_TEST

variable to

local

, because the default is using the release values

sparse-window-1536

04/28/2022, 1:05 AM

final compose:

Copy code

version: "3.5"

services:
  sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    container_name: flyte-sandbox
    privileged: true
    volumes:
      - ../..:/root
    entrypoint: >
      bash -c "
        apk add --update py-pip &&
        pip install yq &&
        yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
        tini flyte-entrypoint.sh
      "
    environment:
      - FLYTE_TEST=local
    ports:
      - "30081:30081"         # Console at <http://localhost:30081/console>
      - "30082:30082"
      - "30084:30084"
      - "30086:30086"
      - "30087:30087"
      - "30088:30088"

❤️ 1

acceptable-policeman-57188

04/28/2022, 6:46 PM

thank you again for sharing!

174 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.