flaky-engineer-56191
06/15/2023, 9:48 AMflytectl demo start
and flytectl sandbox start
? For context, I am NOT using Docker Desktop. I am using Docker Engine + Colima. As such, my docker host is in another path. To change it, I ran the following command as per @curved-easter-24577' suggestion:
export DOCKER_HOST= .......
flytectl config init
When I ran flytectl sandbox start
, it worked perfectly fine. But if I ran flytectl demo start
instead, my workflow failed. Any idea what's happening under the hood? I also tried to use my own docker image when running flytectl sandbox start
but turns out it did not spin up local registry. Any work around?thankful-minister-83577
demo start
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
demo start
ux is betterfreezing-boots-56761
flaky-engineer-56191
06/15/2023, 3:45 PMtar: Removing leading `/' from member names
From times to times, it will result in a failure and produce sigkill(9) which usually happens because the machine is running out of memory. But, I used the same machine when running sandbox start
.flaky-engineer-56191
06/15/2023, 3:47 PMCalledProcessError: Command '['pyflyte-execute', '--inputs',
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/inputs.pb', '--output-prefix',
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ffe430eb7d4f34ccba>
6f/n0/data/0', '--raw-output-data-prefix',
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0>', '--checkpoint-path',
'<s3://my-s3-bucket/data/5h/ffe430eb7d4f34ccba6f-n0-0/_flytecheckpoints>',
'--prev-checkpoint', '""', '--dynamic-addl-distro',
'<s3://my-s3-bucket/flytesnacks/development/LUGCKJRICUAR26UPBIJ7DJZVAE======/scri>
pt_mode.tar.gz', '--dynamic-dest-dir', '/root', '--resolver',
'flytekit.core.python_auto_container.default_task_resolver', '--',
'task-module', 'workflow', 'task-name', 'get_data']' died with <Signals.SIGKILL:
9>.
freezing-boots-56761
flaky-engineer-56191
06/15/2023, 4:04 PMflaky-engineer-56191
06/15/2023, 4:17 PMfreezing-boots-56761
freezing-boots-56761
freezing-boots-56761
flaky-engineer-56191
06/15/2023, 5:11 PMflaky-engineer-56191
06/15/2023, 5:13 PMfreezing-boots-56761
kubectl get pods -A
to list all the podsfreezing-boots-56761
kubectl describe pod <podname> -n <namespace>
to describe itfreezing-boots-56761
flaky-engineer-56191
06/15/2023, 5:48 PMExit Code: 1
Started: Thu, 15 Jun 2023 17:12:07 +0100
Finished: Thu, 15 Jun 2023 17:12:08 +0100
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 200Mi
Requests:
cpu: 2
memory: 200Mi
Environment:
freezing-boots-56761
freezing-boots-56761
flytectl sandbox start
?freezing-boots-56761
thankful-minister-83577
thankful-minister-83577
freezing-boots-56761
freezing-boots-56761
~/.flyte/sandbox/config.yaml
and do flytectl demo reload
task_resources:
defaults:
cpu: "0"
memory: "0"
limits:
cpu: "10"
flaky-engineer-56191
06/16/2023, 8:39 AMflaky-engineer-56191
06/16/2023, 8:43 AMalbertwibowo@Alberts-MacBook-Pro test-aw-flyte % flytectl demo reload
Error:
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check
ERRO[0000]
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check src="main.go:13"
freezing-boots-56761
flaky-engineer-56191
06/16/2023, 2:40 PMflytectl demo reload
. One other thing that I tried was to do these steps:
1. flytectl demo start
2. modify sandbox-config.yaml
3. export SANDBOX_CONFIG = ~/.../config-sandbox.yaml
4. run my workflow
This also did not work.thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
domain: development
project: flytesnacks
defaults:
cpu: "2"
memory: "1Gi"
limits:
cpu: "2"
memory: "2Gi"
write that to a file then
flytectl update task-resource-attribute --attrFile yourfile.yaml
flaky-engineer-56191
06/16/2023, 10:37 PMflaky-engineer-56191
06/16/2023, 10:48 PMthankful-minister-83577
thankful-minister-83577
high-accountant-32689
06/16/2023, 11:54 PMhigh-accountant-32689
06/17/2023, 12:06 AMfreezing-boots-56761
freezing-boots-56761
flytectl sandbox start
is running with this config:
task_resources:
defaults:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
gpu: 5
freezing-boots-56761
freezing-boots-56761
flytectl sandbox start
is using this build.freezing-boots-56761
> less -F ~/.flyte/sandbox/config.yaml
task_resources:
defaults:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
then: flytectl demo reload
freezing-boots-56761
freezing-boots-56761
flytectl demo start
too though. wdyt @thankful-minister-83577 @high-accountant-32689