many-exabyte-15566
06/03/2022, 10:35 PMflytectl sandbox start
) on one machine, and I'm trying to run an example wf from different machine (I've updated endpoint
in ~/.flyte/config.yaml
). pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 21
succeds and I get a link to the console. However when inspecting the execution I see this error:
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f31016626d79e40b2a4c-n0-0] terminated with exit code (1). Reason [Error]. Message:
_execute(cmd=cmd, s3_cfg=self.s3_cfg)
File "/usr/local/lib/python3.8/site-packages/flytekit/extras/persistence/s3_awscli.py", line 51, in _update_cmd_config_and_execute
return subprocess.check_call(anonymous_cmd, env=env)
File "/usr/local/lib/python3.8/site-packages/flytekit/tools/subprocess.py", line 26, in check_call
raise Exception(
Exception: Called process exited with error code: 1. Stderr dump:
b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 495, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 102, in download_distribution
file_access.get_data(additional_distribution, destination)
File "/usr/local/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/36/flytesnacks/development/VGJSBXLCAE3JIKQYKY2O3N5IMQ======/scriptmode.tar.gz> to . (recursive=False).
Original exception: Called process exited with error code: 1. Stderr dump:
b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
.
freezing-airport-6809
freezing-airport-6809
flytectl demo start
?freezing-airport-6809
many-exabyte-15566
06/04/2022, 12:04 AMdemo start
in case that makes a differencefreezing-airport-6809
localhost:30084
on sandboxfreezing-airport-6809
freezing-airport-6809
many-exabyte-15566
06/04/2022, 12:15 AMdemo start
instead of sandboxfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
many-exabyte-15566
06/04/2022, 8:55 PMpyflyte run --remote
was using local storage (I had admin.endpoint
pointing to remote computer, but storage was local). trying to figure out how to point both to remotemany-exabyte-15566
06/04/2022, 8:59 PMflytectl --admin.endpoint dus-mxl950574s:30081 --storage.connection.endpoint dus-mxl950574s:30084 --storage.type minio config init
but it still produces same default config file:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:30081
authType: Pkce
insecure: true
logger:
show-source: true
level: 0
many-exabyte-15566
06/04/2022, 8:59 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
many-exabyte-15566
06/04/2022, 9:02 PMpyflyte --remote
to talk it to itfreezing-airport-6809
many-exabyte-15566
06/04/2022, 9:02 PMadmin.endpoint
it can connect to remote admin, but it's trying to use local miniofreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
fast register
system. In this, we simply bundle the code and start it in an existing containerfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
Remote Flyte cluster
freezing-airport-6809
freezing-airport-6809
many-exabyte-15566
06/04/2022, 9:11 PMfreezing-airport-6809
flytectl sandbox start
then point your kubectl to the k3s inside the flyte-sandbox docker container -freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
many-exabyte-15566
06/04/2022, 9:12 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809