https://flyte.org logo
Powered by
a

Aleksei Potov

06/03/2022, 10:35 PM
I've spun up a Sandbox cluster (using 
flytectl sandbox start
) on one machine, and I'm trying to run an example wf from different machine (I've updated 
endpoint
in 
~/.flyte/config.yaml
). 
pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 21
succeds and I get a link to the console. However when inspecting the execution I see this error:
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f31016626d79e40b2a4c-n0-0] terminated with exit code (1). Reason [Error]. Message: 
_execute(cmd=cmd, s3_cfg=self.s3_cfg)
  File "/usr/local/lib/python3.8/site-packages/flytekit/extras/persistence/s3_awscli.py", line 51, in _update_cmd_config_and_execute
    return subprocess.check_call(anonymous_cmd, env=env)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/subprocess.py", line 26, in check_call
    raise Exception(
Exception: Called process exited with error code: 1.  Stderr dump:

b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 495, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 102, in download_distribution
    file_access.get_data(additional_distribution, destination)
  File "/usr/local/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/36/flytesnacks/development/VGJSBXLCAE3JIKQYKY2O3N5IMQ======/scriptmode.tar.gz> to . (recursive=False).

Original exception: Called process exited with error code: 1.  Stderr dump:

b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
.
k

Ketan (kumare3)

06/03/2022, 11:41 PM
that is werid?
Also you prefer sandbox over 
flytectl demo start
?
should work, you can always open up the minio terminal
a

Aleksei Potov

06/04/2022, 12:04 AM
let me try with 
demo start
in case that makes a difference
k

Ketan (kumare3)

06/04/2022, 12:05 AM
so you can check minio on 
localhost:30084
on sandbox
@Aleksei Potov I can help with some question maybe on a VC, but, I am going to be picking up my daughter soon
cc @Yuvraj / @Prafulla Mahindrakar if you folks are around
a

Aleksei Potov

06/04/2022, 12:15 AM
Looks like it worked when using 
demo start
instead of sandbox
k

Ketan (kumare3)

06/04/2022, 12:15 AM
hmm that is weird
i think maybe minio is just getting throttled
so in demo/sandbox it uses minio as local s3 clone
a

Aleksei Potov

06/04/2022, 8:55 PM
ok, problem was that 
pyflyte run --remote
was using local storage (I had 
admin.endpoint
pointing to remote computer, but storage was local). trying to figure out how to point both to remote
not sure how I'm supposed to configure it though, here's command I'm using: 
flytectl --admin.endpoint dus-mxl950574s:30081 --storage.connection.endpoint dus-mxl950574s:30084 --storage.type minio config init
but it still produces same default config file:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
what I'm doing wrong here?
k

Ketan (kumare3)

06/04/2022, 9:00 PM
hmmm so flytectl does not need storage config anymore
Let me try and understand your setup
you have flyte deployed on some machine?
and then you want flytectl to connect to that machine from another machine?
a

Aleksei Potov

06/04/2022, 9:02 PM
I have remote machine running Flyte in a sandbox, I want 
pyflyte --remote
to talk it to it
k

Ketan (kumare3)

06/04/2022, 9:02 PM
ok
a

Aleksei Potov

06/04/2022, 9:02 PM
if I change 
admin.endpoint
it can connect to remote admin, but it's trying to use local minio
k

Ketan (kumare3)

06/04/2022, 9:03 PM
ya
i got it now
i do not think this is clearly documented (as this was never done :))
but let me look and tell you the configuration
so the way pyflyte run --remote works by default is that flyteadmin returns a signedurl for the content md5 and then pyflyte simply uploads there Code her e- https://github.com/flyteorg/flyteadmin/blob/6e03bd4ac2ad3bb950e8fa1ed64f4bc5c6058bcd/dataproxy/service.go#L72
problem, signed url, returns for the configured endpoint of minio
which is localhost right?
if you have deployed to a static ip, I think we can replace this config with your static ip and then it will do the right thing
unfortunately you cannot do it on client side
another option is for now, do not use the 
fast register
system. In this, we simply bundle the code and start it in an existing container
but Flyte does support your own containers and default registration process
and then 
Remote Flyte cluster
let me know if that helps
and if we want faster iteration, we will have to update the config
a

Aleksei Potov

06/04/2022, 9:11 PM
nice, I will try that. for now forward localhost:30084 to the remote:30084 seems to do the trick 😄
❤️ 1
k

Ketan (kumare3)

06/04/2022, 9:11 PM
and if you are running 
flytectl sandbox start
then point your kubectl to the k3s inside the flyte-sandbox docker container -
ohh ya
i think thats ok for now
when you get a prod setup
this wont be the case
and do not worry I am here to help 😄
a

Aleksei Potov

06/04/2022, 9:12 PM
yah, I get that. for now we want to have a minimal integration setup for a couple of people
k

Ketan (kumare3)

06/04/2022, 9:13 PM
yup makes sense
i think we will improve this thing for your setup scenario
thank you for understanding
29 Views
Powered by