i am trying to save a model plot to wandb using so...
# ask-the-communitya
andrewm4894
03/04/2024, 12:46 PMi am trying to save a model plot to wandb using something like this:
Copy code
with tempfile.NamedTemporaryFile(delete=False, suffix=".png") as tmpfile:
plot_model(model, to_file=tmpfile.name, show_shapes=True, show_layer_names=True)
wandb.log({f"{prefix}Architecture{suffix}": wandb.Image(tmpfile.name)})
os.remove(tmpfile.name) Copy code
/pillow/pillow/PIL/Image.py", line 3305, in open
raise UnidentifiedImageError(msg)
Message:
cannot identify image file '/opt/spark/work-dir/tmpf5ir4qhc.png'
User error.andrewm4894
03/04/2024, 12:49 PMp.s. when i run the workflow locally via a notebook it works fine - only when its registered and run by flyte it seems to not be able to find the file
andrewm4894
03/04/2024, 12:53 PMactually the error above was when i set
dir='.'NamedTemporaryFile() Copy code
cannot identify image file '/tmp/tmp5lh79axa.png'l
L godlike
03/04/2024, 12:58 PMDo you mean thay you want to do this operation in a pod?
L godlike
03/04/2024, 12:59 PMcause the task might work in the pod
a
andrewm4894
03/04/2024, 1:02 PMat moment its just a function happening within a task (task the trains the model - just want to also log the arch plot to wandb).
so i think that means yes i want to do it in a pod, the pod that is about to try train the model in the current task i am running from
l
L godlike
03/04/2024, 1:03 PMfor temp files, I am not familiar with that
L godlike
03/04/2024, 1:03 PMmaybe is a bug, cc @Kevin Su
a
andrewm4894
03/04/2024, 1:05 PMps i tired to use buffer objects to try get around it and avoid need for tempfile but seems that plot_model() wont work with that - if i do that i get
Copy code
TypeError: expected str, bytes or os.PathLike object, not _io.BytesIOl
L godlike
03/04/2024, 1:06 PMwould you like to provide a workflow code for us to reproduce?
also give us which image did you specify, thank you
L godlike
03/04/2024, 1:06 PMjust a basic example can be helpful'
L godlike
03/04/2024, 1:07 PMI am wondering that maybe this is a case about the permission in the pod
L godlike
03/04/2024, 1:07 PMmaybe
/tmpa
andrewm4894
03/04/2024, 1:07 PMhmm actually - might be a packaging issue on my end - i was sure we had graphviz but maybe dont have pydot
l
L godlike
03/04/2024, 1:08 PM Copy code
FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
RUN pip install -U git+<https://github.com/flyteorg/flytekit.git@c9972bf023d10bef3aefa9fa6763c6b35aec5693>L godlike
03/04/2024, 1:08 PMyou can install all dependency you want in a dockerfile, and need to include flytekit
a
andrewm4894
03/04/2024, 1:08 PMill double check that its all available where its running - but looks like maybe false alarm
l
L godlike
03/04/2024, 1:08 PMand specify the image
L godlike
03/04/2024, 1:08 PMok, thanks
a
andrewm4894
03/04/2024, 1:08 PMyep cool - will talk to the infra guys about how i can do that internally
andrewm4894
03/04/2024, 1:09 PMcheers
10 Views