https://flyte.org logo
Powered by
d

Dylan Wilder

05/02/2022, 5:16 PM
sorry for so much spam today 😅 for context this is all around upgrading to 1.0. Getting the following issue when registering with pyflyte. see 🧵
Copy code
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flytekit/clis/sdk_in_container/serialize.py", line 147, in workflows
    config_file=ctx.obj.get(constants.CTX_CONFIG_FILE, None),
  File "/usr/local/lib/python3.7/dist-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flytekit/clis/sdk_in_container/serialize.py", line 69, in serialize_all
    serialize_to_folder(pkgs, serialization_settings, local_source_root, folder)
  File "/usr/local/lib/python3.7/dist-packages/flytekit/tools/repo.py", line 62, in serialize_to_folder
    persist_registrable_entities(loaded_entities, folder)
  File "/usr/local/lib/python3.7/dist-packages/flytekit/tools/serialize_helpers.py", line 132, in persist_registrable_entities
    fname = _os.path.join(folder, fname)
  File "/usr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
is pyflyte not compatible with 1.0?
that'd be a blocker for us
e

Eduardo Apolinario (eapolinario)

05/02/2022, 6:53 PM
@Dylan Wilder, you should be able to serialize with pyflyte. Can you share more details? How are you running this? Is this a regression (i.e. the exact command used to user prior to 1.0)?
d

Dylan Wilder

05/02/2022, 6:57 PM
yea nothing changed except upgrading flytekit
command
Copy code
docker run \
  -e FLYTE_INTERNAL_IMAGE \
  -e FLYTE_PLATFORM_AUTH \
  -e FLYTE_PLATFORM_INSECURE \
  -e FLYTE_CREDENTIALS_AUTH_MODE \
  -e FLYTE_CREDENTIALS_COMMAND \
  -e FLYTE_CREDENTIALS_AUTHORIZATION_METADATA_KEY \
  -e FLYTE_PLATFORM_URL \
  -e GOOGLE_APPLICATION_CREDENTIALS \
  -e GOOGLE_APPLICATION_CREDENTIALS_DIR \
  --mount type=bind,source=${SOURCE_GOOGLE_APPLICATION_CREDENTIALS},target=${GOOGLE_APPLICATION_CREDENTIALS_DIR} \
  --network=host \
  $FLYTE_INTERNAL_IMAGE bash -ce "
  gcloud config set component_manager/disable_update_check true &&
  pyflyte -c '${CONFIG_FILE}' --pkgs vivaldi_sandbox --pkgs onemodel --pkgs workflows serialize --image ${FLYTE_INTERNAL_IMAGE} workflows
  flyte-cli register-files -p ${PROJECT} -d ${DOMAIN} -v ${VERSION} -h ${HOST} -o ${DATA} -s default *.pb
e

Eduardo Apolinario (eapolinario)

05/02/2022, 6:59 PM
cc: @Yee
cool, thanks. And just to speed up the investigation, what version of flytekit were you running before?
d

Dylan Wilder

05/02/2022, 7:02 PM
0.23.1 i think
wait is this an issue with the flyte config file?
(looking more closely at the error)
k

Ketan (kumare3)

05/02/2022, 7:39 PM
config is no longer required or used
it has been so for a while
would love to see what you have in the config
d

Dylan Wilder

05/02/2022, 8:13 PM
is that the issue?
namely i'm looking at this line in the exception
Copy code
config_file=ctx.obj.get(constants.CTX_CONFIG_FILE, None),
we have this i guess
Copy code
[resources]
default_cpu_request = 1
default_cpu_limit = 2
default_memory_request = 2Gi
default_memory_limit = 3Gi
solved, actually needed a 
-f pb_output
y

Yee

05/02/2022, 10:56 PM
🤦 sorry dylan… we need to improve our error messaging for sure.
d

Dylan Wilder

05/02/2022, 11:57 PM
No problem. I see the bug in error handling and it's an easy one to make. But now flyte-cli fails to fork something and bricks my terminal :(
moved to flytectl and got it to work
ok, but now pyflyte is prefixing all the workflows with the current directories name...
k

Ketan (kumare3)

05/03/2022, 12:50 AM
Ya, it is localizing them
So that you can run them From anywhere
d

Dylan Wilder

05/03/2022, 12:51 AM
sorry what does that mean?
not the end of the world, but names are important for consumption downstream
so is a breaking change, also noticed that this doesn't happen when i run from the docker container (previous method)
wait it also appears to break at runtime. one-model is the prefix it added
Copy code
ModuleNotFoundError: No module named 'one-model'
k

Ketan (kumare3)

05/03/2022, 1:56 AM
It should happen in all cases
This is not good
It was part of the release notes
Would have time to sync, how this breaks
We're you running serialize from inside a python package somewhere?
Or were you following the default way
y

Yee

05/03/2022, 2:21 AM
dylan - let’s hop on a call later today or tomorrow
and interesting you say that… the same thing happened to me, hung on a fork - we will have to investigate further. I tracked it down to the line of code that opens a web browser tab for the callback to work. The forking is happening normally, but because the browser tab isn’t opening, it appears hung (we should also add some better logging).
i wasn’t able to reproduce it consistently. and things worked again after i switched over my default osx browser to firefox and seemed to continue to work after i switched it back.
i’m still not sure if it’s directly flytekit related because at the time, running 
jupyter lab
which normally opens a tab as well also did not work and I had to manually copy the link it gave me into the browser.
d

Dylan Wilder

05/03/2022, 12:20 PM
yea i might be able to today
does today work?
y

Yee

05/03/2022, 5:25 PM
can we do 1pm my time?
d

Dylan Wilder

05/03/2022, 7:04 PM
sure that works
@Yee?
y

Yee

05/03/2022, 7:59 PM
gimme two mins
e

Eduardo Apolinario (eapolinario)

05/03/2022, 8:01 PM
Hey, can I also join?
y

Yee

05/03/2022, 8:02 PM
d

Dylan Wilder

05/03/2022, 8:05 PM
joining
7 Views
Powered by