Aleksei Potov
06/03/2022, 10:35 PMflytectl sandbox start
) on one machine, and I'm trying to run an example wf from different machine (I've updated endpoint
in ~/.flyte/config.yaml
). pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 21
succeds and I get a link to the console. However when inspecting the execution I see this error:
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f31016626d79e40b2a4c-n0-0] terminated with exit code (1). Reason [Error]. Message:
_execute(cmd=cmd, s3_cfg=self.s3_cfg)
File "/usr/local/lib/python3.8/site-packages/flytekit/extras/persistence/s3_awscli.py", line 51, in _update_cmd_config_and_execute
return subprocess.check_call(anonymous_cmd, env=env)
File "/usr/local/lib/python3.8/site-packages/flytekit/tools/subprocess.py", line 26, in check_call
raise Exception(
Exception: Called process exited with error code: 1. Stderr dump:
b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 495, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 102, in download_distribution
file_access.get_data(additional_distribution, destination)
File "/usr/local/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/36/flytesnacks/development/VGJSBXLCAE3JIKQYKY2O3N5IMQ======/scriptmode.tar.gz> to . (recursive=False).
Original exception: Called process exited with error code: 1. Stderr dump:
b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
.
Ketan (kumare3)
flytectl demo start
?Aleksei Potov
06/04/2022, 12:04 AMdemo start
in case that makes a differenceKetan (kumare3)
localhost:30084
on sandboxAleksei Potov
06/04/2022, 12:15 AMdemo start
instead of sandboxKetan (kumare3)
Aleksei Potov
06/04/2022, 8:55 PMpyflyte run --remote
was using local storage (I had admin.endpoint
pointing to remote computer, but storage was local). trying to figure out how to point both to remoteflytectl --admin.endpoint dus-mxl950574s:30081 --storage.connection.endpoint dus-mxl950574s:30084 --storage.type minio config init
but it still produces same default config file:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:30081
authType: Pkce
insecure: true
logger:
show-source: true
level: 0
Ketan (kumare3)
Aleksei Potov
06/04/2022, 9:02 PMpyflyte --remote
to talk it to itKetan (kumare3)
Aleksei Potov
06/04/2022, 9:02 PMadmin.endpoint
it can connect to remote admin, but it's trying to use local minioKetan (kumare3)
fast register
system. In this, we simply bundle the code and start it in an existing containerRemote Flyte cluster
Aleksei Potov
06/04/2022, 9:11 PMKetan (kumare3)
flytectl sandbox start
then point your kubectl to the k3s inside the flyte-sandbox docker container -Aleksei Potov
06/04/2022, 9:12 PMKetan (kumare3)