#3338 [BUG] - Spark Executors can't access the code from the Driver with Fast Registration Issue created by TheoLacour-Artefact Describe the bug Spark Executors can't access the code from the Spark Driver when new code is packaged with Fast Registration for Spark Tasks that use a

map

function. The Spark Executors can only access the code from the image it pulled. Expected behavior The Spark Executors should use the same code as the Spark Driver - which is the code that was pushed using Fast Registration. Additional context to reproduce 1. Following this exemple : use the code from https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html , building a Docker image like this one https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/index.html#step-2-environment-setup 2.

pyflyte --pkgs my_code --image my_image --force

3.

flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.1

4. Go to the Flyte Console, run the workflow : it works as expected, the output of

my_spark

is close to 3.14 5. Then, change the code from the function

f

in step 1: now,

f

always returns 1. 6. I updated my code but I don't need to update my environnement, so I can use Fast Registration :

pyflyte --pkgs my_code --image my_image --force --fast

7.

flytectl register files --project my_project --domain development --archive flyte-package.tgz --outputLocationPrefix <s3://my_bucket> --k8sServiceAccount my_account --version 0.0.2

8. Go to the Flyte Console, run the workflow's new version : the output of

my_spark

is close to 3.14, but I was expecting it to be 4. Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte