aloof-painting-18735
06/14/2023, 9:42 AMpyflyte
on the cluster and triggered the run like this:
pyflyte run -remote dbx_example_existing_cluster.py my_databricks_job
The Databricks job is started, but failing with this error:
TypeError: loader must define exec_module()
glamorous-carpet-83516
06/14/2023, 9:50 AMglamorous-carpet-83516
06/14/2023, 9:50 AMglamorous-carpet-83516
06/14/2023, 9:51 AMaloof-painting-18735
06/14/2023, 10:03 AMaloof-painting-18735
06/14/2023, 10:04 AMaloof-painting-18735
06/14/2023, 10:04 AMFor demonstration purposes, I saved time on setup by employing an existing cluster that already had the necessary Python dependencies installed. In a production environment, you can choose to configure a job cluster using a custom Docker image based on Databricks' base build, ensuring that all necessary dependencies are present.
aloof-painting-18735
06/14/2023, 10:05 AMaloof-painting-18735
06/14/2023, 10:06 AMpyflyte
package and copied the actual code
to /databricks/driver/databricks/
aloof-painting-18735
06/14/2023, 10:07 AMbillowy-winter-86593
06/14/2023, 1:01 PMaloof-painting-18735
06/14/2023, 1:03 PMaloof-painting-18735
06/14/2023, 1:05 PMaloof-painting-18735
06/14/2023, 1:05 PMaloof-painting-18735
06/14/2023, 1:05 PMaloof-painting-18735
06/14/2023, 1:06 PMpyflyte
package as I did?billowy-winter-86593
06/14/2023, 1:11 PMaloof-painting-18735
06/14/2023, 1:19 PMaloof-painting-18735
06/14/2023, 2:07 PMflytekit
package, create a notebook with these import statements:
from flytekit.core.workflow import ImperativeWorkflow as Workflow
from flytekit.core.workflow import WorkflowFailurePolicy, reference_workflow, workflow
from flytekit.deck import Deck
from flytekit.image_spec import ImageSpec
from flytekit.loggers import logger
and run it, I got the same issue:aloof-painting-18735
06/14/2023, 2:09 PMaloof-painting-18735
06/14/2023, 2:09 PMaloof-painting-18735
06/14/2023, 2:09 PMaloof-painting-18735
06/14/2023, 2:10 PMaloof-painting-18735
06/14/2023, 2:11 PMaloof-painting-18735
06/14/2023, 2:11 PMDBR 11.3LTS
and flytekit==1.3.0b4
aloof-painting-18735
06/14/2023, 2:18 PMModuleNotFoundError: No module named 'flytekit.image_spec'
aloof-painting-18735
06/14/2023, 2:18 PMDBR 11.3LTS
and flytekit==1.7.0
aloof-painting-18735
06/14/2023, 2:28 PMTypeError: loader must define exec_module()
again, so this issue might be specific to flytekit==1.7.0