Théo LACOUR
02/09/2023, 2:26 PMf
, and then use pyflyte --pkgs path.to.code package --image my_image --force --fast
to package the code and then upload and run the new version of the workflow, we notice that the behavior f
is the same as before.
As I understand it now, this could be because the Spark executors still have the previous code. Does this work as intended ?
TL;DR : can we use --fast
to avoid building / pushing Docker images for Spark tasks ?Ketan (kumare3)
pyflyte run
directlypyflyte run --remote --image <http://ghcr.io/flyteorg/flytecookbook:k8s_spark-43585feeccabc8a48452dc6838426f3acf4c6a9d|ghcr.io/flyteorg/flytecookbook:k8s_spark-43585feeccabc8a48452dc6838426f3acf4c6a9d> pyspark_pi.py my_spark --triggered_date now
Théo LACOUR
02/10/2023, 9:56 AMflytectl register files --archive flyte-package.tgz etc.
with a correct project, domain and service account. Then I use the console to run the workflow.
The behavior I am expecting : the new code packaged in my .tgz file should be used by the Spark executors, since I used the --fast
tag. What I observed : the Spark driver code is updated, but the Spark executors code is the one from the image, not the one from the .tgz file.
I wonder if this is expected behavior, and if it is, how can I register a workflow with new code without having to re-build a Docker image ?Ketan (kumare3)
Evan Sadler
02/10/2023, 3:09 PMKetan (kumare3)
Evan Sadler
02/10/2023, 3:17 PMpyflyte register -*-destination-dir .*
Théo LACOUR
02/10/2023, 3:51 PMEvan Sadler
02/10/2023, 3:52 PM/root
or whatever I had set it to. Good luck!Théo LACOUR
02/13/2023, 1:17 PM--destinationDirectory
in my command `flytectl register etc.`which has the same effect as pyflyte register --destination-dir etc.
. It did not solve my problem (as my code did actually run in /root
) so I might write a GitHub issue later, unless this works 'as intended' or as a limitation of Spark (i.e. Spark executors should be expected to pull the image and use it 'as is' instead of using the updated code).Tyler Su
02/15/2023, 11:11 PMThéo LACOUR
02/16/2023, 9:23 AMYini Gao
04/12/2023, 10:54 AM