https://flyte.org
Join Slack
i am trying to save a model plot to wandb using something like this: ```with tempfile.NamedTemporar...
# flyte-support
f
i am trying to save a model plot to wandb using something like this:
Copy code
with tempfile.NamedTemporaryFile(delete=False, suffix=".png") as tmpfile:
    plot_model(model, to_file=tmpfile.name, show_shapes=True, show_layer_names=True)
    wandb.log({f"{prefix}Architecture{suffix}": wandb.Image(tmpfile.name)})
os.remove(tmpfile.name)
it works apart from when run by Flyte where i get:
Copy code
/pillow/pillow/PIL/Image.py", line 3305, in open
        raise UnidentifiedImageError(msg)

Message:

    cannot identify image file '/opt/spark/work-dir/tmpf5ir4qhc.png'

User error.
Just wondering - Is there anything specific to Flyte around working with temp files?
👀 1
p.s. when i run the workflow locally via a notebook it works fine - only when its registered and run by flyte it seems to not be able to find the file
actually the error above was when i set 
dir='.'
in 
NamedTemporaryFile()
when i leave it out it tries to use a tmp dir but has same issue:
Copy code
cannot identify image file '/tmp/tmp5lh79axa.png'
d
Do you mean thay you want to do this operation in a pod?
cause the task might work in the pod
f
at moment its just a function happening within a task (task the trains the model - just want to also log the arch plot to wandb). so i think that means yes i want to do it in a pod, the pod that is about to try train the model in the current task i am running from
d
for temp files, I am not familiar with that
maybe is a bug, cc @glamorous-carpet-83516
f
ps i tired to use buffer objects to try get around it and avoid need for tempfile but seems that plot_model() wont work with that - if i do that i get
Copy code
TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO
d
would you like to provide a workflow code for us to reproduce? also give us which image did you specify, thank you
just a basic example can be helpful'
I am wondering that maybe this is a case about the permission in the pod
maybe 
/tmp
is not writable
f
hmm actually - might be a packaging issue on my end - i was sure we had graphviz but maybe dont have pydot
d
Copy code
FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y

RUN pip install -U git+<https://github.com/flyteorg/flytekit.git@c9972bf023d10bef3aefa9fa6763c6b35aec5693>
you can install all dependency you want in a dockerfile, and need to include flytekit
f
ill double check that its all available where its running - but looks like maybe false alarm
d
and specify the image
ok, thanks
f
yep cool - will talk to the infra guys about how i can do that internally
cheers
🙏 1
🤩 1
27 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by