https://flyte.org logo
#ask-the-community
Title
# ask-the-community
h

Han

04/26/2022, 8:24 PM
Couple of problems I ran into while I try to go through this env setup tutorial. 1. I am seeing
<http://googletagmanager.com|googletagmanager.com>
and
<http://gstatic.com|gstatic.com>
for flytectl local sandbox dashboard links. Are they necessary? 2. What's the purpose of
pyflyte
? 3. cont'd
3. Running the pyflyte --remote example locally on Windows WSL2,.
Copy code
:~/flytesnacks/cookbook$ pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f2066c1781e1b4b31bd9> to see execution in the console.
But on the console I see pod failed.
Copy code
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.

[f2066c1781e1b4b31bd9-n0-0] terminated with exit code (137). Reason [OOMKilled]. Message: 
tar: Removing leading `/' from member names
This is using
flytectl sandbox start
.
flytectl demo start
doesn't work for me. Version 0.5.21.
Running the same
pyflyte --remote
command on our shared linux server I am getting
Copy code
pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
E0426 20:09:01.812198758   57838 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0426 20:09:01.814035626   57838 <http://ssl_transport_security.cc:1504]|ssl_transport_security.cc:1504]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
{"asctime": "2022-04-26 20:09:01,820", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651003741.820656500\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651003741.820655223\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
{"asctime": "2022-04-26 20:09:02,021", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1651003742.021633228\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3128,\"referenced_errors\":[{\"created\":\"@1651003742.021631791\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":163,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
Traceback (most recent call last):
  File "/venv/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 152, in run
    wf = remote.register_script(
  File "/venv/lib/python3.8/site-packages/flytekit/remote/remote.py", line 517, in register_script
    upload_location, md5_version = fast_register_single_script(
  File "/venv/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 118, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/venv/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/venv/lib/python3.8/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1651003742.422810939","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1651003742.422810939","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
y

Yee

04/26/2022, 10:12 PM
hey - when you run
flytectl demo start
what error do you get?
re
pyflyte
- it’s a python click command that has a variety of commands beneath it. these are most importantly: 1.
package
which goes through the local python code files, and compiles the objects into Flyte protobufs. 2.
run
which does the same thing, but only for one file (currently) and additionally also sends those protofbufs over to the Flyte backend service (Admin). It does so by uploading that file to S3/GCS/Azure blob storage, and creating a task definition that relies on a prebuilt default image, modifying the command to pull back down that file before running.
re the error message… which Flyte version are you running on the backend?
it looks like it is failing on the
CreateUploadLocation
request, which is a new endpoint we added to handle step 2 (
run
) above, when the uploading to the blob store happens.
for 1. could you give us a bit more information? where are you seeing this?
can you send a screenshot?
oh and re the oom… the default example doesn’t set any memory requests, which means they’re being set by admin on the backend using defaults. would it be possible to bump up the defaults a bit?
k

katrina

04/26/2022, 10:29 PM
@Yee didn't we bump the admin defaults b/c of ooms on M1s? but regardless shouldn't they just work out of the box for all use cases now?
y

Yee

04/26/2022, 10:47 PM
yeah i’m not sure what the configuration is, just wanted to check.
h

Han

04/27/2022, 1:56 AM
Okay. Restarted everything using
Copy code
flytectl demo start
and run
Copy code
pyflyte run --remote core/flyte_basics/hello_world.py:my_wf
Was abvle to get outputs on dashboard.
What's the diference between
flytectl demo start
vs
flytectl sandbox start
?
RE: #1, it shows up on my js blocker
I am using
flytectl
version 0.5.21. And flytekit 0.32.3 as specified in
flytesnacks/core/requirements.txt
And one other question, if I have an workflow that I can execute locally, i.e.
Copy code
python workflow.py
The
@workflow
decorated workflow function takes a few arguments. How do I pass in arguments using
pyflyte
? And how do I run the same workflow in the sandbox/demo environment?
k

Ketan (kumare3)

04/27/2022, 3:41 AM
Using pyflyte you can type —help and that will Show all arguments
👍 1
s

Samhita Alla

04/27/2022, 5:19 AM
What’s the diference between
flytectl demo start
vs
flytectl sandbox start
?
flytectl demo
uses single binary to spin up the Flyte sandbox. To get started quickly in a limited testing environment,
flytectl demo start
is preferred to
flytectl sandbox start
.
And how do I run the same workflow in the sandbox/demo environment?
Please use the
--remote
flag. Make sure
~/.flyte/config.yaml
is set up. This has to happen automatically on installing flytectl.
re: https://flyte-org.slack.com/archives/CP2HDHKE1/p1651024686417919?thread_ts=1651004643.081319&amp;cid=CP2HDHKE1, googletagmanager is used for analytics and gstatic for static content.
h

Han

04/27/2022, 5:49 AM
Okay. And
@dynamic
decoractor will allow for dynamic workflows using native python instead of, say,
conditional()
type of logic branching DSL, correct? Will there be an workflow DAG w/
@dynamic
decorators?
Also, when I do
pyflyte --remote
, how do I ensure that the packages exist in the demo env? aka, im getting ModuleNotFoundError for some of my imports
flytectl demo
uses single binary to spin up the Flyte sandbox. To get started quickly in a limited testing environment,
flytectl demo start
is preferred to
flytectl sandbox start
.
Okay, so for proof of concept,
fyltectl demo start
. Under what condition is
flytectl sandbox start
preferred?
k

Ketan (kumare3)

04/27/2022, 6:02 AM
Demo start is a full drop in replacement
👍 1
One place where sandbox is better if you want to modify the helm chart etc today
That will change soon
s

Samhita Alla

04/27/2022, 6:15 AM
And
@dynamic
decoractor will allow for dynamic workflows using native python instead of, say,
conditional()
type of logic branching DSL, correct?
@dynamic
is useful when you cannot determine the DAG structure beforehand.
conditional()
is similar to the
if..else
construct, meaning, it switches b/w the conditions but doesn’t dynamically determine the DAG structure.
Will there be an workflow DAG w/
@dynamic
decorators?
Do you mean if a workflow DAG is possible?
Also, when I do
pyflyte --remote
, how do I ensure that the packages exist in the demo env? aka, im getting ModuleNotFoundError for some of my imports
pyflyte run --remote
uses a default image bundled with flytekit, which contains numpy, pandas, and flytekit. If you want to install additional dependencies, you’ll have to build your Docker image and send that as an argument to pyflyte using the
--image
flag.
k

Ketan (kumare3)

04/27/2022, 1:32 PM
This image has to be pushed, else you can follow the image build instructions- cc @Yee
👍 1
91 Views