#4877 [BUG] Lazy load the `pyspark.ml` module breaks on Spark clusters Issue created by eapolinario Describe the bug After installing flytekitplugins-spark==1.10.3 we get this error on Spark tasks:

Traceback (most recent call last):
  File "/opt/venv/bin/entrypoint.py", line 16, in <module>
    from flytekit.configuration import (
  File "/opt/venv/lib/python3.9/site-packages/flytekit/__init__.py", line 305, in <module>
    load_implicit_plugins()
  File "/opt/venv/lib/python3.9/site-packages/flytekit/__init__.py", line 301, in load_implicit_plugins
    p.load()
  File "/opt/venv/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 184, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/opt/venv/lib/python3.9/site-packages/flytekitplugins/spark/__init__.py", line 21, in <module>
    from .pyspark_transformers import PySparkPipelineModelTransformer
  File "/opt/venv/lib/python3.9/site-packages/flytekitplugins/spark/pyspark_transformers.py", line 7, in <module>
    pyspark_ml = lazy_module("<http://pyspark.ml|pyspark.ml>")
  File "/opt/venv/lib/python3.9/site-packages/flytekit/lazy_import/lazy_module.py", line 41, in lazy_module
    loader = importlib.util.LazyLoader(spec.loader)
  File "/usr/lib/python3.9/importlib/util.py", line 282, in __init__
    self.__check_eager_loader(loader)
  File "/usr/lib/python3.9/importlib/util.py", line 273, in __check_eager_loader
    raise TypeError('loader must define exec_module()')

This doesn't repro locally (i.e. when only installing

flytekitplugins-spark==1.10.3

and interacting with a python interpreter or running local tasks). Expected behavior Lazy-loading modules should work in all cases, including when running Spark tasks. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte