Hi! This is such a great project, i'm having problems when i add "from flytekitplugins.spark import Spark" to my *.py after running (remote way, im not having problems using 'pyflyte run *.py my_wf) the process is complete, but the wokflow appears failed with the following message from web UI: [1/1] currentAttempt done. Last Error: USER::trap>", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/root/flyte.py", line 5, in from flytekitplugins.spark import Spark ModuleNotFoundError: No module named 'flytekitplugins.spark' Traceback (most recent call last): File "/usr/local/bin/pyflyte-fast-execute", line 8, in sys.exit(fast_execute_task_cmd()) ...

You need to install flytekit plugins spark in the container

great! i have it installed in my local python env, how can i install it in the container?

@wide-helmet-70417 you have to write a dockerfile to build an image. btw, you could try to use image spec. Here is an example to build image for spark task by using image spec. https://github.com/flyteorg/flytekit/pull/1616

Thanks @glamorous-carpet-83516, i tried image spec and it works with run, but not with run --remote.

you have to install envd plugin as well. pip install flytekitplugins-envd==1.6.1

I think we should include envd in default? And then allow it to be overridden?

+1 to including it in the flytekit library.

@glamorous-carpet-83516 i installed it sucesfully and got this 'failed to build the imageSpec' - autorization failed

Have you specified the registry? I think you should authenticate to dockerhub from the CLI. https://docs.docker.com/docker-hub/access-tokens/

@tall-lock-23197 im able to run the spark example shared by kevin now with --remote flag! but got the following error in the web UI: [1/1] currentAttempt done. Last Error: USER:core.py760 in invoke │ │ │ │ ❱ 760 │ │ │ │ return __callback(*args, **kwargs) │ │ │ │ /usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:508 in │ │ fast_execute_task_cmd │ │ │ │ ❱ 508 │ subprocess.run(cmd, check=True) │ │ │ │ /usr/local/lib/python3.10/subprocess.py:526 in run │ │ │ │ ❱ 526 │ │ │ raise CalledProcessError(retcode, process.args, │ ╰──────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['pyflyte-execute', '--inputs', 's3://my-s3-bucket/metadata/propeller/flytesnacks-development-f91eee73c35a9498e9 c8/n0/data/inputs.pb', '--output-prefix', 's3://my-s3-bucket/metadata/propeller/flytesnacks-development-f91eee73c35a9498e9 c8/n0/data/0', '--raw-output-data-prefix', 's3://my-s3-bucket/data/dv/f91eee73c35a9498e9c8-n0-0', '--checkpoint-path', 's3://my-s3-bucket/data/dv/f91eee73c35a9498e9c8-n0-0/_flytecheckpoints', '--prev-checkpoint', '""', '--dynamic-addl-distro', 's3://my-s3-bucket/flytesnacks/development/EFP2R3DRITTXTRCHKYZCLZWAZU======/scri pt_mode.tar.gz', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flyte3', 'task-name', 'hello_spark']' returned non-zero exit status 1.

This is weird. It is not understanding that something has changed- cc @glamorous-carpet-83516 / @high-accountant-32689 this is unexpected @wide-helmet-70417 is this after you use the new Imagespec

@freezing-airport-6809, yes, im using spark_image = ImageSpec(registry="<mydockerhubuser>"), and running it with flytekit==1.6.1 and flytekitplugins-envd==1.6.1

im able to run the spark example shared by kevin now with --remote flag! but got the following error in the web UI:

Would you be able to fetch the k8s log? https://docs.flyte.org/en/latest/community/troubleshoot.html#troubleshoot

kubectl get pods -n flytesnacks-development
kubectl logs <pod-name> -n flytesnacks-development

@tall-lock-23197 Sure!, this is the output

tar: Removing leading `/' from member names

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/pyflyte-execute:8 in <module>                                 │
│                                                                              │
│ ❱ 8 │   sys.exit(execute_task_cmd())                                         │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1130 in __call__       │
│                                                                              │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                             │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1055 in main           │
│                                                                              │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                             │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1404 in invoke         │
│                                                                              │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)            │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:760 in invoke          │
│                                                                              │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                    │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:471 in    │
│ execute_task_cmd                                                             │
│                                                                              │
│ ❱ 471 │   _execute_task(                                                     │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py:160 in │
│ system_entry_point                                                           │
│                                                                              │
│ ❱ 160 │   │   │   │   return wrapped(*args, **kwargs)                        │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:347 in    │
│ _execute_task                                                                │
│                                                                              │
│ ❱ 347 │   │   _task_def = resolver_obj.load_task(loader_args=resolver_args)  │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/core/utils.py:295 in        │
│ wrapper                                                                      │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/core/python_auto_container. │
│ py:235 in load_task                                                          │
│                                                                              │
│ ❱ 235 │   │   task_module = importlib.import_module(name=task_module)  # typ │
│                                                                              │
│ /usr/local/lib/python3.10/importlib/__init__.py:126 in import_module         │
│                                                                              │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│ in _gcd_import:1050                                                          │
│ in _find_and_load:1027                                                       │
│ in _find_and_load_unlocked:1006                                              │
│ in _load_unlocked:688                                                        │
│ in exec_module:883                                                           │
│ in _call_with_frames_removed:241                                             │
│                                                                              │
│ /root/flyte3.py:7 in <module>                                                │
│                                                                              │
│ ❱  7 from flytekitplugins.spark import Spark                                 │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'flytekitplugins.spark'
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/pyflyte-fast-execute:8 in <module>                            │
│                                                                              │
│ ❱ 8 │   sys.exit(fast_execute_task_cmd())                                    │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1130 in __call__       │
│                                                                              │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                             │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1055 in main           │
│                                                                              │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                             │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1404 in invoke         │
│                                                                              │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)            │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/click/core.py:760 in invoke          │
│                                                                              │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                    │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:508 in    │
│ fast_execute_task_cmd                                                        │
│                                                                              │
│ ❱ 508 │   subprocess.run(cmd, check=True)                                    │
│                                                                              │
│ /usr/local/lib/python3.10/subprocess.py:526 in run                           │
│                                                                              │
│ ❱  526 │   │   │   raise CalledProcessError(retcode, process.args,           │
╰──────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['pyflyte-execute', '--inputs', 
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f52f3d40c3cf849a89>
5a/n0/data/inputs.pb', '--output-prefix', 
'<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-f52f3d40c3cf849a89>
5a/n0/data/0', '--raw-output-data-prefix', 
'<s3://my-s3-bucket/data/rr/f52f3d40c3cf849a895a-n0-0>', '--checkpoint-path', 
'<s3://my-s3-bucket/data/rr/f52f3d40c3cf849a895a-n0-0/_flytecheckpoints>', 
'--prev-checkpoint', '""', '--dynamic-addl-distro', 
'<s3://my-s3-bucket/flytesnacks/development/U67AW4X3WQLPZXY2HBRFM7GDWY======/scri>
pt_mode.tar.gz', '--dynamic-dest-dir', '/root', '--resolver', 
'flytekit.core.python_auto_container.default_task_resolver', '--', 
'task-module', 'flyte3', 'task-name', 'hello_spark']' returned non-zero exit 
status 1.

sorry, actually. you can’t use fast-register if spark task is in the workflow because spark worker won’t download the code. try to use

pyflyte register …

instead.

@glamorous-carpet-83516 why is that?

when I use fast-register, only driver downloaded code