https://flyte.org logo
Powered by
#announcements
Title
# announcements
m

Matheus Moreno

04/27/2022, 6:50 PM
Hi, everyone! Quick question: is it possible to raise the memory limit on a Flyte sandbox? A member of my team is trying to test their system locally, but is being greeted with an OOMKilled error.
k

katrina

04/27/2022, 7:45 PM
hi @Matheus Moreno do you mean the memory limit for an individual task or for docker itself?
m

Matheus Moreno

04/27/2022, 8:43 PM
for an individual task. if I try to request more than 1Gi, I get an error on the registration process
k

katrina

04/27/2022, 8:58 PM
ah, for that we need to update the task limits in the flyteadmin configmap. can i ask how your teammate is starting up flytesandbox?
m

Matheus Moreno

04/27/2022, 9:43 PM
she's using docker directly (actually, docker-compose). can it be configured from there?
k

katrina

04/27/2022, 9:59 PM
ok, so we need to modify the flyteadmin config map request limits here. also sorry, what do you mean by 
docker directly
?
m

Matheus Moreno

04/27/2022, 10:01 PM
oh, i just meant that she's not using flytectl, but actually using 
docker run ...
to start the server
but how can I edit this file? bypassing the entrypoint of the sandbox to first edit it?
k

katrina

04/27/2022, 10:03 PM
can you share the full docker run command?
m

Matheus Moreno

04/27/2022, 10:03 PM
Copy code
docker create --name flyte-sandbox \
    --privileged \
    --mount type=bind,src=$PWD,dst=/root \
    -p 30081:30081 -p 30082:30082 -p 30084:30084 -p 30086-30088:30086-30088 \
    <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>

docker run flyte-sandbox            # Start sandbox
the compose is
Copy code
version: "3.5"

services:
  sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    container_name: flyte-sandbox
    privileged: true
    volumes:
      - ../..:/root
    ports:
      - "30081:30081"         # Console at <http://localhost:30081/console>
      - "30082:30082"
      - "30084:30084"
      - "30086:30086"
      - "30087:30087"
      - "30088:30088"
(it's in a subsubdirectory)
k

katrina

04/27/2022, 11:03 PM
do you have a way to modify the kubernetes resources brought up by the sandbox?
m

Matheus Moreno

04/27/2022, 11:07 PM
trying to do that rn
I'm trying to add a short script before the entrypoint command to replace the value on values.yaml
🙏 1
this is a VERY extreme go horse solution but it works, I think. I installed pip and yq to make the substitution (I did not want to spend hours trying to make the right sed command):
Copy code
entrypoint: >
      bash -c "
        apk add --update py-pip &&
        pip install yq &&
        yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
        tini flyte-entrypoint.sh
      "
k

katrina

04/28/2022, 12:12 AM
amazing!! thanks for sharing
m

Matheus Moreno

04/28/2022, 12:35 AM
welp, it did not work. I changed the values on both 
/flyteorg/share/flyte/values.yaml
and 
/flyteorg/share/flyte-core/values.yaml
and this error still happens during registration:
Copy code
Error: rpc error: code = InvalidArgument desc = Requested MEMORY default [4Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
i confirmed that the substitution process worked by entering the sandbox container and inspecting these files. 
.flyte.configmap.task_resource_defaults.task_resources.limits.memory
is configured to 4Gi to both
what could be the problem? 😕
nevermind, fixed it! I had to specify the 
FLYTE_TEST
variable to 
local
, because the default is using the release values
final compose:
Copy code
version: "3.5"

services:
  sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    container_name: flyte-sandbox
    privileged: true
    volumes:
      - ../..:/root
    entrypoint: >
      bash -c "
        apk add --update py-pip &&
        pip install yq &&
        yq -i -y '.flyte.configmap.task_resource_defaults.task_resources.limits.memory = \"4Gi\"' /flyteorg/share/flyte/values.yaml &&
        tini flyte-entrypoint.sh
      "
    environment:
      - FLYTE_TEST=local
    ports:
      - "30081:30081"         # Console at <http://localhost:30081/console>
      - "30082:30082"
      - "30084:30084"
      - "30086:30086"
      - "30087:30087"
      - "30088:30088"
❤️ 1
k

katrina

04/28/2022, 6:46 PM
thank you again for sharing!
11 Views
Powered by