Albert Wibowo
06/15/2023, 9:48 AMflytectl demo start
and flytectl sandbox start
? For context, I am NOT using Docker Desktop. I am using Docker Engine + Colima. As such, my docker host is in another path. To change it, I ran the following command as per @Marti Jorda Roca' suggestion:
export DOCKER_HOST= .......
flytectl config init
When I ran flytectl sandbox start
, it worked perfectly fine. But if I ran flytectl demo start
instead, my workflow failed. Any idea what's happening under the hood? I also tried to use my own docker image when running flytectl sandbox start
but turns out it did not spin up local registry. Any work around?Yee
demo start
demo start
ux is betterjeev
Albert Wibowo
06/15/2023, 3:45 PMtar: Removing leading `/' from member names
From times to times, it will result in a failure and produce sigkill(9) which usually happens because the machine is running out of memory. But, I used the same machine when running sandbox start
.CalledProcessError: Command '['pyflyte-execute', '--inputs',
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/inputs.pb', '--output-prefix',
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/0', '--raw-output-data-prefix',
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0>', '--checkpoint-path',
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0/_flytecheckpoints>',
'--prev-checkpoint', '""', '--dynamic-addl-distro',
'<s3://my-s3-bucket/flytesnacks/development/LUGCKJRICUAR26UPBIJ7DJZVAE======/scri>
pt_mode.tar.gz', '--dynamic-dest-dir', '/root', '--resolver',
'flytekit.core.python_auto_container.default_task_resolver', '--',
'task-module', 'workflow', 'task-name', 'get_data']' died with <Signals.SIGKILL:
9>.
jeev
Albert Wibowo
06/15/2023, 4:04 PMjeev
Albert Wibowo
06/15/2023, 5:11 PMjeev
kubectl get pods -A
to list all the podskubectl describe pod <podname> -n <namespace>
to describe itAlbert Wibowo
06/15/2023, 5:48 PMExit Code: 1
Started: Thu, 15 Jun 2023 17:12:07 +0100
Finished: Thu, 15 Jun 2023 17:12:08 +0100
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 200Mi
Requests:
cpu: 2
memory: 200Mi
Environment:
jeev
flytectl sandbox start
?Yee
jeev
~/.flyte/sandbox/config.yaml
and do flytectl demo reload
task_resources:
defaults:
cpu: "0"
memory: "0"
limits:
cpu: "10"
Albert Wibowo
06/16/2023, 8:39 AMalbertwibowo@Alberts-MacBook-Pro test-aw-flyte % flytectl demo reload
Error:
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check
ERRO[0000]
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check src="main.go:13"
jeev
Albert Wibowo
06/16/2023, 2:40 PMflytectl demo reload
. One other thing that I tried was to do these steps:
1. flytectl demo start
2. modify sandbox-config.yaml
3. export SANDBOX_CONFIG = ~/.../config-sandbox.yaml
4. run my workflow
This also did not work.Yee
domain: development
project: flytesnacks
defaults:
cpu: "2"
memory: "1Gi"
limits:
cpu: "2"
memory: "2Gi"
write that to a file then
flytectl update task-resource-attribute --attrFile yourfile.yaml
Albert Wibowo
06/16/2023, 10:37 PMYee
Eduardo Apolinario (eapolinario)
06/16/2023, 11:54 PMjeev
flytectl sandbox start
is running with this config:
task_resources:
defaults:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
gpu: 5
flytectl sandbox start
is using this build.> less -F ~/.flyte/sandbox/config.yaml
task_resources:
defaults:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
then: flytectl demo reload
flytectl demo start
too though. wdyt @Yee @Eduardo Apolinario (eapolinario)