Yash Panchwatkar
10/02/2022, 12:41 PMYash Panchwatkar
10/02/2022, 12:42 PMYash Panchwatkar
10/02/2022, 12:42 PMRobin Eklund
10/03/2022, 9:10 AMTony Vec
10/03/2022, 1:44 PMSecretsManager
within tests with the goal of not accidentally using secrets in /etc/secrets
. We have a way of doing this now (I’ll add what we have in 🧵) but was hoping to see if there’s a standard way of doing thisGeorge D. Torres
10/03/2022, 6:33 PMcache=True
). Is there a better way? What about for workflows that do return values?Krishna Agarwal
10/04/2022, 2:35 AMOpen AIMP
10/04/2022, 4:31 AMSamhita Alla
10/04/2022, 11:20 AMRobin Eklund
10/04/2022, 12:51 PMfrom flytekit import task, workflow
@task
def task_a() -> None:
print("task_a")
@task
def task_b() -> None:
print("task_b")
@task
def task_c() -> None:
print("task_c")
@workflow
def my_wf() -> None:
task_a()
task_b()
task_c()
this workflow would run task_a, task_b and task_c in parallel. But i want them to run sequentially.Geoff Salmon
10/04/2022, 4:18 PMRobin Kahlow
10/04/2022, 5:44 PMfield
defaults on dataclasses? Dataclasses can't have list / dict / set defaults (https://stackoverflow.com/q/52063759) and you have to resort to something like str_list: List[str] = field(default_factory=lambda: ["a", "b"])
instead, but on Flyte console when launching a workflow it just shows an empty list for me.Padma Priya M
10/05/2022, 5:58 AMcron_lp = LaunchPlan.get_or_create(
name="cron_scheduler",
workflow=scheduler_workflow,
schedule=CronSchedule(
schedule="*/1 * * * *",
),
default_inputs={'f': 4.5},
)
Robert Everson
10/05/2022, 9:45 PMHamza Azelmat
10/06/2022, 11:58 AMNo such file or directory: '/v3/f25eaabfb23924c25946-n1-0/d1a84d1c0cea75f237d21a70dcde6810/model1-out.ipynb'
Ian Calvert
10/06/2022, 3:50 PMLouis DiNatale
10/06/2022, 5:18 PMFrank Shen
10/06/2022, 6:35 PMfrom lib.subwf import my_subwf
I can run the flyte flow successfully in local mode.
However, it errored when running with --remote in the demo flyte cluster I started on my PC.
The error msg is:
ModuleNotFoundError: No module named 'lib'
Frank Shen
10/06/2022, 6:36 PMFrank Shen
10/06/2022, 6:36 PMAdwitiya
10/07/2022, 1:21 PMSrishti Mehra
10/07/2022, 3:21 PMclass LGBMDatasetTransformer(extend.TypeTransformer[lgb.Dataset]):
_TYPE_INFO = flytekit.BlobType(
format="binary", dimensionality=flytekit.BlobType.BlobDimensionality.MULTIPART
)
def __init__(self) -> None:
super(LGBMDatasetTransformer, self).__init__(name="lgbdataset-transform", t=lgb.Dataset)
def get_literal_type(self, t: Type[lgb.Dataset]) -> flytekit.LiteralType:
return flytekit.LiteralType(blob=self._TYPE_INFO)
def to_literal(
self,
ctx: flytekit.FlyteContext,
python_val: lgb.Dataset,
python_type: Type[lgb.Dataset],
expected: flytekit.LiteralType,
) -> flytekit.Literal:
"""
Convert from the given python type object ``lgb.Dataset`` to the Literal representation.
"""
# Step 1: Upload all the data into a remote place recommended by Flyte
local_file = ctx.file_access.get_random_local_path() + ".data"
lgb.Dataset.save_binary(python_val, local_file)
remote_dir = ctx.file_access.get_random_remote_directory()
ctx.file_access.upload_directory(local_file, remote_dir)
# Step 2: Return a pointer to this remote_dir in the form of a Literal
return flytekit.Literal(
scalar=flytekit.Scalar(
blob=flytekit.Blob(
uri=remote_dir, metadata=flytekit.BlobMetadata(type=self._TYPE_INFO)
)
)
)
def to_python_value(
self,
ctx: flytekit.FlyteContext,
lv: flytekit.Literal,
expected_python_type: Type[lgb.Dataset],
) -> lgb.Dataset:
"""
In this method, we want to be able to re-hydrate the custom object from Flyte Literal value.
"""
# Step 1: Download remote data locally
local_dir = ctx.file_access.get_random_local_directory()
ctx.file_access.download_directory(lv.scalar.blob.uri, local_dir)
# Step 2: Create the ``lgb.Dataset`` object
return lgb.Dataset.construct(lgb.Dataset(local_dir))
extend.TypeEngine.register(LGBMDatasetTransformer())
fei sun
10/08/2022, 3:10 AMYash Panchwatkar
10/08/2022, 7:13 AMYash Panchwatkar
10/08/2022, 7:15 AMTolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned flyte/minio-5745464bb-zthw8 to ip-172-31-31-189.ap-south-1.compute.internal
Normal Pulling 16m kubelet Pulling image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>"
Normal Pulled 16m kubelet Successfully pulled image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" in 13.816089274s
Normal Created 14m (x5 over 16m) kubelet Created container minio
Normal Started 14m (x5 over 16m) kubelet Started container minio
Normal Pulled 14m (x4 over 16m) kubelet Container image "<http://ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0|ecr.flyte.org/bitnami/minio:2021.10.13-debian-10-r0>" already present on machine
Warning BackOff 85s (x61 over 15m) kubelet Back-off restarting failed container
and result from logs
[root@ip-172-31-31-189 ~]# kubectl logs minio-5745464bb-zthw8 -n flyte
07:15:01.89
07:15:01.90 Welcome to the Bitnami minio container
07:15:01.90 Subscribe to project updates by watching <https://github.com/bitnami/bitnami-docker-minio>
07:15:01.90 Submit issues and feature requests at <https://github.com/bitnami/bitnami-docker-minio/issues>
07:15:01.90
07:15:01.90 INFO ==> ** Starting MinIO setup **
07:15:02.01 INFO ==> Starting MinIO in background...
07:15:12.01 INFO ==> Adding local Minio host to 'mc' configuration...
07:15:12.41 INFO ==> MinIO is already stopped...
0xsapphir3
10/09/2022, 8:43 AMKetan (kumare3)
10/09/2022, 3:00 PMGanesh Patil
10/10/2022, 3:13 AMKeshav Sharma
10/10/2022, 4:43 AMlaunch plans
with different set of fixed inputs
for an existing workflow
.
I am following flytekit documentation to fetch a workflow and create a launchplan for it.
The following in the code snippet from above docs:
from flytekit import LaunchPlan
flyte_workflow = remote.fetch_workflow(
name="my_workflow", version="v1", project="flytesnacks", domain="development"
)
launch_plan = LaunchPlan.get_or_create(name="my_launch_plan", workflow=flyte_workflow)
Creating a launchplan fails with the following error:
File "/Users/.../lib/python3.7/site-packages/flytekit/core/launch_plan.py", line 143, in create
native_types=workflow.python_interface.inputs,
AttributeError: 'NoneType' object has no attribute 'inputs'
python_interface
attribute of workflow object returned by fetch_workflow
is None, although the workflow is a valid one and when i register a launchplan using "register files" in cli, it works.
flytekit version: 1.1.0
flyte helm chart version: 1.1.0
Any help appreciated.Sathish kumar Venkatesan
10/10/2022, 8:41 AMdetails = "launch plan with different structure already exists with id resource_type:LAUNCH_PLAN project:"cloudops-max-flyte-demo" domain:"development" name:"snowflake_cron_scheduled_lp" version:"v1"
Sathish kumar Venkatesan
10/10/2022, 8:41 AMdetails = "launch plan with different structure already exists with id resource_type:LAUNCH_PLAN project:"cloudops-max-flyte-demo" domain:"development" name:"snowflake_cron_scheduled_lp" version:"v1"
Samhita Alla
10/10/2022, 10:49 AMSathish kumar Venkatesan
10/10/2022, 10:51 AMSamhita Alla
10/10/2022, 10:53 AMSathish kumar Venkatesan
10/10/2022, 11:20 AMSamhita Alla
10/10/2022, 12:09 PMKetan (kumare3)
10/10/2022, 1:50 PM