Fabio Grätz
03/07/2023, 11:51 AMnn.Module
) need to be serialized into blob storage. Instead of a bucket uri, a /tmp/flyte-xyz
path is shown in the console:
I expect that my flytekit/remote config is not correct and would appreciate some help to figure out where exactly. Thanks 🙏
More in the 🧵from flytekit import task, workflow
from torch import nn
class Config: # enforce pickle transport
def __init__(self):
self.a = 1
@task
def train(cfg: Config, model: nn.Module):
print(cfg)
@workflow
def wf():
train(cfg=Config(), model=nn.Linear(10, 10))
to_literal
method of the pickle transformer (here). It results in:
execution state: ExecutionState(mode=None, working_dir=PosixPath('/tmp/flyte-z15pwqet/sandbox/local_flytekit'), engine_dir='/tmp/flyte-z15pwqet/sandbox/local_flytekit/engine_dir', branch_eval_mode=None, user_space_params=<flytekit.core.context_manager.ExecutionParameters object at 0x7f7a445d6520>)
working_dir
and the engine_dir
of the ExecutionState
are the same as in the Flyte console screenshot.pyflyte register --project sandbox --version ... --destination-dir /home/flyte --image ... test_pickle.py
~/.flyte/config.yaml
contains this:
admin:
...
logger:
show-source: true
level: 0
storage:
type: stow
stow:
kind: google
config:
json: ""
project_id: <project_id>
scopes: <https://www.googleapis.com/auth/devstorage.read_write>
Samhita Alla
--raw-data-prefix
in your pyflyte register
command to the s3 bucket?Fabio Grätz
03/09/2023, 2:11 PM--raw-data-prefix
(with a gs://
uri) doesn’t have an effect on this, the console still shows /tmp/flyte-
inputs 🤔--raw-data-prefix
have changed this? In the links I pasted here, it looks like the /tmp/flyte-
paths are hardcoded.Samhita Alla
--raw-data-prefix
should have changed this. What's the raw output prefix in the propeller config?
cc @Eduardo Apolinario (eapolinario)Fabio Grätz
03/10/2023, 8:33 AMrawoutput-prefix: gs://...
Samhita Alla
Eduardo Apolinario (eapolinario)
03/15/2023, 10:14 PMConfig
and nn.Module
in the definition of the workflow in your example) locally.
Essentially I'm saying:
import pandas as pd
from flytekit import task, workflow
from torch import nn
class Config: # enforce pickle transport
def __init__(self):
self.a = 1
@task
def get_default_config() -> Config:
return Config()
@task
def get_default_model() -> nn.Module:
return nn.Linear(10, 10)
@task
def train(cfg: Config, model: nn.Module):
print(cfg)
@workflow
def wf():
train(cfg=get_default_config(), model=get_default_model())
Fabio Grätz
03/16/2023, 5:24 PM/tmp/…
dir will not be available in the cluster and it takes some amount of understanding what is happening to figure out why this fails.
Is there any way in the type transformer to identify from the ctx
that we are in registration mode and not local execution?Eduardo Apolinario (eapolinario)
03/16/2023, 10:22 PMFabio Grätz
03/17/2023, 9:00 AM