Sanjay Chouhan
09/28/2022, 9:39 AMHampus Rosvall
09/28/2022, 12:41 PM{flyte/sandbox}.config
and what are best practices for usage as a production config? Referencing this sandbox.configAnthony
09/28/2022, 12:53 PMpreproc_and_split
step was executed successfully:
pyflyte-execute
--inputs
<s3://my-s3-bucket/metadata/propeller/flyte-anti-fraud-ml-development-a27rchl5z9ndpw297nk8/n0/data/inputs.pb>
--output-prefix
<s3://my-s3-bucket/metadata/propeller/flyte-anti-fraud-ml-development-a27rchl5z9ndpw297nk8/n0/data/0>
--raw-output-data-prefix
<s3://my-s3-bucket/vo/a27rchl5z9ndpw297nk8-n0-0>
--checkpoint-path
<s3://my-s3-bucket/vo/a27rchl5z9ndpw297nk8-n0-0/_flytecheckpoints>
--prev-checkpoint
""
--resolver
flytekit.core.python_auto_container.default_task_resolver
--
task-module
app.workflow
task-name
preproc_and_split
On the output one should expect a small train dataset with 50k records.
In Nods allocation i see a sufficient mem available.
But then the first task has been succeeded i see an eternal hang in this step and flyte don’t produce next executions according to the workflow.
task_resource_defaults
conf is the next:
task_resource_defaults.yaml: |
task_resources:
defaults:
cpu: 1
memory: 3000Mi
storage: 200Mi
limits:
cpu: 5
gpu: 1
memory: 8Gi
storage: 500Mi
I have one task that generates a dataclasses instances on the exit and another task should takes these classes as input params:
@workflow
def main_flow() -> Forecast:
"""
Main Flyte WorkFlow consisting of three tasks:
- @preproc_and_split
- @train_xgboost_clf
- @get_predictions
"""
<http://logger.info|logger.info>(log="#START -- START Raw Preprocessing and Splitting", timestamp=None)
train_cls, target_cls = preproc_and_split()
<http://logger.info|logger.info>(log="#START -- START Initialize Boosting Params", timestamp=None)
saved_mpath = train_xgboost_clf(
feat_cls=train_cls,
target_cls=target_cls,
xgb_params=xgb_params,
cust_metric=BoostingCustMetric
)
Where
def preproc_and_split() -> Tuple[Fraud_Raw_PostProc_Data_Class, Fraud_Raw_Target_Data_Class]:
Any advices why I faced this behaviour?Hampus Rosvall
09/28/2022, 2:09 PMAugie Palacios
09/28/2022, 7:15 PM/console
to /my/url/console
but the UI still looks for the js assests in /console/assets/*
. This is just the beginning though, will eventually need to prepend a path to /api/v1/projects
as well so it doesn't interfere with our endpointsFrank Shen
09/28/2022, 7:24 PMDylan Wilder
09/28/2022, 9:12 PMKim Junil
09/29/2022, 1:36 AMChandramoulee K V
09/29/2022, 6:42 AMflytectl get task --project flytesnacks --domain development {taskname} --latest --execFile exec_spec.yaml
and this modified the exec_spec.yaml
and it looks like this:
iamRoleARN: ""
inputs: {}
kubeServiceAcct: ""
targetDomain: ""
targetProject: ""
task: {taskname}
version: "1.0"
and i tried to create an execution with the following command :
flytectl create execution --execFile exec_spec.yaml -p flytesnacks -d development --targetProject flytesnacks
and it throws an error like:
Error: rpc error: code = Internal desc = failed to create workflow in propeller json: error calling MarshalJSON for type *v1alpha1.Inputs: Marshal called with nil
KS Tarun
09/29/2022, 6:44 AMModuleNotFoundError
, though that module was installed when creating the docker image.
And this is getting resolved when I pip install
that module in the instance.
So, please clarify whether on running a workflow, would flyte propeller look for the dependencies in the specified docker container or it tries to use the dependencies from the server it is running in ?Sanjay Chouhan
09/29/2022, 8:29 AMHampus Rosvall
09/29/2022, 3:19 PMlp.py
but it is maybe the cleanest solution to this problem?
(.venv) ☁ flyte [main] tree .
.
├── <http://in_container.mk|in_container.mk>
├── launchplans
│ └── lp.py
└── workflows
├── wf.py
James Evers
09/29/2022, 7:33 PMContainerTask
. Both a small test that I wrote myself and the cookbook example here fail when run locally. The cookbook example runs successfully when its run remotely. Any help here is appreciated!karthikraj
09/30/2022, 2:52 AMfrom flytekitplugins.snowflake import SnowflakeConfig, SnowflakeTask
.
I need to do the same thing with my custom plugin too. -> from flytekitplugins.mycustomtask import MyCustomTask
or similarly.Arshak Ulubabyan
09/30/2022, 8:34 AMSanjay Chouhan
09/30/2022, 11:27 AMpyflyte --config config.yaml register test1.py --version 1.0.1
I have created the code as mentioned in the doc, https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/multi_images.html#
The config.yaml is,
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///##############-16##<http://26454.us-west-1.elb.amazonaws.com:80|26454.us-west-1.elb.amazonaws.com:80>
authType: Pkce
insecure: true
logger:
show-source: true
level: 0
images:
trainer: <http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>
predictor: moulee31/sample:1.0
The error is,
Traceback (most recent call last):
File "/home/sanjaychouhan/.local/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/register.py", line 174, in register
registerable_entities = load_packages_and_modules(
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 225, in load_packages_and_modules
registrable_entities = serialize(pkgs_and_modules, ss, str(project_root), options)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/repo.py", line 54, in serialize
registrable_entities = get_registrable_entities(ctx, options=options)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/serialize_helpers.py", line 75, in get_registrable_entities
get_serializable(new_api_serializable_entities, ctx.serialization_settings, entity, options=options)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 578, in get_serializable
cp_entity = get_serializable_task(entity_mapping, settings, entity)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/tools/translator.py", line 173, in get_serializable_task
container = entity.get_container(settings)
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 164, in get_container
image=get_registerable_container_image(self.container_image, settings.image_config),
File "/home/sanjaychouhan/.local/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 235, in get_registerable_container_image
raise AssertionError(f"Image Config with name {name} not found in the configuration")
AssertionError: Image Config with name trainer not found in the configuration
It was working with pyflyte run command,
pyflyte --config config.yaml run --remote test1.py test_workflow
Sanjay Chouhan
09/30/2022, 1:45 PMJames Evers
09/30/2022, 3:09 PMdocker system prune -a --volumes
frees up a sometimes surprising amount of space (i ran the mnist tutorial a few times and pruning the volumes revealed that it had used ~14GB on my machine). is there a way to reduce/mitigate this or is it just a necessary result of saving workflow executions?Nicholas LoFaso
09/30/2022, 4:14 PMFailed to find the Resource with name: dpp-default/g20210730154015-yjww-n0-0-dn4-0-dn108-0. Error: pods \"g20210730154015-yjww-n0-0-dn4-0-dn108-0\" not found
Flyte restarts the task and it succeeds on the 2nd or 3rd try, but this is obviously wasted work. I’m curious if this is FlytePropeller needing more CPU/Memory to accommodate or if we are overwhelming the k8s metadata server. Any thoughts would be appreciatedHank Fanchiu
09/30/2022, 5:59 PMHampus Rosvall
10/01/2022, 8:07 AMpyflyte register
and how do adopt it in our set up. When I package code using pyflyte package
I usually provide the dot-delineated python packages to operate on i.e.,
pyflyte --pkgs flyte.workflows package --image $REGISTRY/$REPO:$TAG -o $(PACKAGE_OUTPUT_DIR)/package.tgz
When I run pyflyte register
I would like to do the same in order to fast-register to the same workflow i.e.,
pyflyte register flyte.workflows \
--version=$VERSION \
--image $REGISTRY/$REPO:$TAG \
--project $PROJECT \
--domain $DOMAIN
However running that command yields
Usage: pyflyte register [OPTIONS] [PACKAGE_OR_MODULE]...
Try 'pyflyte register --help' for help.
Error: Invalid value for '[PACKAGE_OR_MODULE]...': Path 'flyte.workflows' does not exist.
gmake: *** [Makefile:42: flyte-fast-register] Error 2
Do I need to provide the path to the workflow? If I pass flyte/workflows/wf.py
to pyflyte register
my workflow is registered under <http://wf.wf|wf.wf>
instead of <http://flyte.workflows.wf.wf|flyte.workflows.wf.wf>
as with pyflyte package
(.venv) tree .
.
├── Dockerfile
├── Makefile
├── flyte
│ ├── __init__.py
│ ├── <http://in_container.mk|in_container.mk>
│ ├── launchplans
│ │ └── lp.py
│ └── workflows
│ ├── __init__.py
│ └── wf.py
Taeef Najib
10/01/2022, 11:04 AM# Creating the Multiple Linear Regression model
def build_model(model, X_train: pd.DataFrame, y_train: pd.DataFrame):
# Use the following values: model = LinearRegression(), X_train = X_train, y_train = y_train
reg = model
reg.fit(X_train, y_train)
return reg
I’m using int
, float
, pd.DataFrame
, np.ndarray
etc. as my type hints.
My question is what type hint do I use for the argument model
and since it is returning the model, what type hint should I use for reg
?Hridya Agrawal
10/01/2022, 6:41 PMHridya Agrawal
10/01/2022, 6:41 PMHridya Agrawal
10/01/2022, 6:42 PMSanjiv Anand
10/02/2022, 3:45 AMYash Panchwatkar
10/02/2022, 10:35 AMYash Panchwatkar
10/02/2022, 10:35 AMYash Panchwatkar
10/02/2022, 10:35 AMYash Panchwatkar
10/02/2022, 10:47 AMYash Panchwatkar
10/02/2022, 10:47 AM