https://flyte.org
Join Slack
Hello, what's the difference between `flytectl demo start` and `flytectl sandbox start`? For context...
# flyte-support
f
Hello, what's the difference between 
flytectl demo start
and 
flytectl sandbox start
? For context, I am NOT using Docker Desktop. I am using Docker Engine + Colima. As such, my docker host is in another path. To change it, I ran the following command as per @curved-easter-24577' suggestion:
Copy code
export DOCKER_HOST= .......
flytectl config init
When I ran 
flytectl sandbox start
, it worked perfectly fine. But if I ran 
flytectl demo start
instead, my workflow failed. Any idea what's happening under the hood? I also tried to use my own docker image when running 
flytectl sandbox start
but turns out it did not spin up local registry. Any work around?
✅ 1
t
we really need to replace the old command. you should use 
demo start
@freezing-boots-56761 should we rename?
👍 1
sandbox start brings up a version of flyte where all the backend components are deployed separately. we realized a while back that for most production installations that is not necessary, so we combined them.
also the 
demo start
ux is better
f
@flaky-engineer-56191 what was the failure in the workflow?
f
Well it just keeps running forever. When I looked at the Kubernetes Logs, the only thing I saw was:
Copy code
tar: Removing leading `/' from member names
From times to times, it will result in a failure and produce sigkill(9) which usually happens because the machine is running out of memory. But, I used the same machine when running 
sandbox start
.
In case the workflow failed:
Copy code
CalledProcessError: Command '['pyflyte-execute', '--inputs', 
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/inputs.pb', '--output-prefix', 
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/0', '--raw-output-data-prefix', 
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0>', '--checkpoint-path', 
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0/_flytecheckpoints>', 
'--prev-checkpoint', '""', '--dynamic-addl-distro', 
'<s3://my-s3-bucket/flytesnacks/development/LUGCKJRICUAR26UPBIJ7DJZVAE======/scri>
pt_mode.tar.gz', '--dynamic-dest-dir', '/root', '--resolver', 
'flytekit.core.python_auto_container.default_task_resolver', '--', 
'task-module', 'workflow', 'task-name', 'get_data']' died with <Signals.SIGKILL:
9>.
f
can you give docker more memory?
f
I can try. I will be back.
@freezing-boots-56761, it does not work. I still got the same error. I gave docker 12 GB of memory with 6 CPUs.
f
is the memory usage unexpected?
@flaky-engineer-56191 can you describe the failed pod and see what the resource requests/limits were?
it might just be getting oomkilled because it was exceeding the task's limit
f
yeah it was unexpected because my workflow is quite simple actually. The workflow is simply loading dataset from scikit-learn and then visualising the data using flytedeck. The weird thing is that if I use docker deskop instead, with the same CPUs and memory dedicated to docker, it works just fine. Also, I am new to docker + kubernetes, How do i describe the failed pods?
So in summary: What works for me • Docker Engine + Colima + flytectl sandbox start • Docker Desktop + flytectl demo start What does not work for me • Docker Engine + Colima + flytectl demo start In all of these cases, I used the same workflow + CPUs + memory
f
kubectl get pods -A
to list all the pods
kubectl describe pod <podname> -n <namespace>
to describe it
thats pretty interesting though. why would a task come up and then get oomkilled?
f
You are right - one of the pods is getting oomkilled. And it looks like the pod is not registering the new CPUs and Memory size it seems.
Copy code
Exit Code:  1
   Started:   Thu, 15 Jun 2023 17:12:07 +0100
   Finished:   Thu, 15 Jun 2023 17:12:08 +0100
  Ready:     False
  Restart Count: 0
  Limits:
   cpu:   2
   memory: 200Mi
  Requests:
   cpu:   2
   memory: 200Mi
  Environment:
f
200Mi seems like very little
is it the same in 
flytectl sandbox start
?
might be a config issue
t
isn’t this the default that’s set in admin?
f
ah i always forget this
try writing this to 
~/.flyte/sandbox/config.yaml
and do 
flytectl demo reload
Copy code
task_resources:
  defaults:
    cpu: "0"
    memory: "0"
  limits:
    cpu: "10"
f
alright will try it
I dont think I can do reload when I edit the config file directly:
Copy code
albertwibowo@Alberts-MacBook-Pro test-aw-flyte % flytectl demo reload
Error: 
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check
ERRO[0000] 
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check src="main.go:13"
f
where did you write the file to?
f
I tried writing it directly to either config.yaml and config-sandbox.yaml. Both gave me the same error when trying to run 
flytectl demo reload
. One other thing that I tried was to do these steps: 1. flytectl demo start 2. modify sandbox-config.yaml 3. export SANDBOX_CONFIG = ~/.../config-sandbox.yaml 4. run my workflow This also did not work.
t
we really need to get rid of the defaults in there.
can you try adding a project/domain level override?
Copy code
domain: development
project: flytesnacks
defaults:
  cpu: "2"
  memory: "1Gi"
limits:
  cpu: "2"
  memory: "2Gi"
write that to a file then
Copy code
flytectl update task-resource-attribute --attrFile yourfile.yaml
f
Yup sure, I'll try it. Thank you so much for being patience.
yes it finally worked! Thanks @thankful-minister-83577 @freezing-boots-56761. I guess I have to do this every time then? How come when I use docker desktop to increase the size of the CPU and memory flyte is able to update the config automatically?
t
@high-accountant-32689 can we fix this?
We just need to extract out the relevant parts of the pr and merge it
h
Sure thing.
Still not clear why this works in the case of the old sandbox. Do we have other overrides in that case?
f
need to test
thx 1
flytectl sandbox start
is running with this config:
Copy code
task_resources:
  defaults:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 2
    memory: 4Gi
    gpu: 5
that would explain
👍 1
haha looked at sandbox-lite and realized that was the wrong one too!! 
flytectl sandbox start
is using this build.
@flaky-engineer-56191: you can just run with this:
Copy code
> less -F ~/.flyte/sandbox/config.yaml
task_resources:
  defaults:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 2
    memory: 4Gi
then: 
flytectl demo reload
🙏 1
that will persist for as long as your config is at that path
those are probably more reasonable defaults for 
flytectl demo start
too though. wdyt @thankful-minister-83577 @high-accountant-32689
👍 1
346 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by