astonishing-lizard-78628
05/22/2022, 2:25 AMv1.0.0
with flytekit==0.26.0
and Spark tasks are fine. I just moved up (last night) to flytekit==1.0.1
with flytekitplugins-spark==1.0.1
(with the same backend system) and now Spark tasks are broken with the error can't open file '/usr/bin/entrypoint.py': [Errno 2] No such file or directory
. I took a look and the old (good) tasks have the following set:
"mainApplicationFile": {
"stringValue": "local:///usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py"
}
and that file exists and looks fine. However, my new (broken) Spark task shows the following:
"mainApplicationFile": {
"stringValue": "local:///usr/bin/entrypoint.py"
},
"executorPath": {
"stringValue": "/usr/bin/python3.8"
}
but I have no /usr/bin/entrypoint.py
in the container (which explains the error message). This seems 99% like a bug, can you take a look? I do have a v1.0.1
Flyte backend system up and running... I'll try the same thing there.astonishing-lizard-78628
05/22/2022, 2:28 AMPYSPARK_DRIVER_PYTHON=python3.8
PYSPARK_PYTHON=python3.8
since I want to use python3.8
(and not the python3
installed in my Bionic container). It doesn't seem like these env variables would affect the issue with mainApplicationFile
, however.astonishing-lizard-78628
05/22/2022, 2:44 AMv1.0.1
. All these have been with flytectl register files
. I'll also switch over to flyte-cli register files
and check if it makes any difference.astonishing-lizard-78628
05/22/2022, 2:56 AMflyte-cli register files
alsoastonishing-lizard-78628
05/22/2022, 2:57 AMflytekit==1.0.1
due to an error setting the wrong mainApplicationFile
path. They were working for me back in flytekit 0.26.0 so something broke them after that.astonishing-lizard-78628
05/22/2022, 2:57 AMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
astonishing-lizard-78628
05/23/2022, 11:22 AMastonishing-lizard-78628
05/23/2022, 11:24 AMpython3.8
installed at /usr/bin/python3.8
.
FYI that since there is also a separate (3.6.x) Python 3 at /usr/bin/python3, I have declared:
PYSPARK_DRIVER_PYTHON=python3.8
PYSPARK_PYTHON=python3.8
environment variables in my container.freezing-airport-6809
astonishing-lizard-78628
05/23/2022, 9:17 PM0.26.0
which is working for us, the path to the Flyte entrypoint.py
file was formed explicitly and correctly hereastonishing-lizard-78628
05/23/2022, 9:19 PM0.26.0
the value of flytekit_virtualenv_root
is '/usr/local/lib/python3.8/dist-packages/flytekit'
which is determined correctly in the code from:
import flyte;
flytekit_install_loc = os.path.abspath(flytekit.__file__)
ctx.obj[CTX_FLYTEKIT_VIRTUALENV_ROOT] = os.path.dirname(flytekit_install_loc)
astonishing-lizard-78628
05/23/2022, 9:24 PMentrypoint
path determination is dropped. It is now calculated here as a default based on the Python interpreter path which is just /usr/bin/python3.8
in my container. Whomever made the change clearly assumed people would only be running pyflyte
commands from a venv and not with the system interpreter.astonishing-lizard-78628
05/23/2022, 9:39 PMpyflyte
executable being called and assume that entrypoint.py
next to that (not sure if this is possible in Python).astonishing-lizard-78628
05/23/2022, 9:40 PMflytekit
package and appending /bin/entrypoint.py
does work robustly.astonishing-lizard-78628
05/23/2022, 9:42 PM/usr/bin/python3 pip install flytekit flytekitplugins-spark
in their container and to try Flyte and that will fail.astonishing-lizard-78628
05/23/2022, 9:48 PMpyflyte register --in-container-virtualenv-root /usr/local/lib/python3.8/dist-packages/flytekit
as a workaround for now. I think that should get Spark tasks working for me again. Thanks for pointing that out! Lol that Miguel wrote it. I looked through our Git and we had actually stopped using that a while back, but we'll use it again.astonishing-lizard-78628
05/24/2022, 1:48 AM--in-container-virtualenv-root
can't be used as a workaround when I'm actually using /usr/bin/python3.8
to install things in my container. The right workaround for the issue is just ln -s /usr/local/bin/entrypoint.py /usr/bin/entrypoint.py
in the container. This fixes it; my Spark jobs are working again.astonishing-lizard-78628
05/24/2022, 1:49 AMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
glamorous-father-92994
05/24/2022, 4:24 PM--in-container-virtualenv-root
stuff off the top of my head