GitHub
03/15/2024, 10:43 PMGenericDynamicLogLinksEnabled
, which is self-explanatory. We then use a dynamic log link guard called "Generic" to guide the addition of generic dynamic log links (what I'm calling these log links defined in tasks - a better name could be in order).
How was this patch tested?
Unit tests and local sandbox.
Setup process
Screenshots
Check all the applicable boxes
☐ I updated the documentation accordingly.
☐ All new and existing tests passed.
☐ All commits are signed-off.
Related PRs
Docs link
flyteorg/flyte
GitHub Actions: build-and-push-sandbox-bundled-image
✅ 29 other checks have passed
29/30 successful checksGitHub
03/15/2024, 10:59 PMAsyncAgentBase
, output_prefix
was removed from the create
function for agents. This is problematic for our use case, and in general seems like it is moving things towards less flexible agents.
We would also like to be able to consume the task execution metadata in our create function if possible, and I may tack that on to this PR later.
What changes were proposed in this pull request?
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
☐ I updated the documentation accordingly.
☐ All new and existing tests passed.
☐ All commits are signed-off.
Related PRs
Docs link
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
03/15/2024, 11:47 PMimport typing
from flytekit import ImageSpec, Resources, task, workflow
commit_hash = "5a57f299c1ae789c9e2f269302becf13ca3b21e8"
ray_plugin = f"git+<https://github.com/flyteorg/flytekit.git@{commit_hash}#subdirectory=plugins/flytekit-ray>"
custom_image = ImageSpec(
name="ray-flyte-plugin",
registry="localhost:30000",
packages=[ray_plugin],
apt_packages=['git', 'wget', 'vim'],
)
if custom_image.is_container():
import ray
from flytekitplugins.ray import HeadNodeConfig, RayJobConfig, WorkerNodeConfig
@ray.remote
def f(x):
print(a)
print(f"ray task get {x}")
return x * x
ray_config = RayJobConfig(
head_node_config=HeadNodeConfig(ray_start_params={"log-color": "True"}),
worker_node_config=[WorkerNodeConfig(group_name="ray-group", replicas=1)],
runtime_env={"pip": []},
)
@task(
task_config=ray_config,
requests=Resources(mem="4Gi", cpu="1"),
container_image=custom_image,
)
def ray_task(n: int) -> typing.List[int]:
futures = [f.remote(i) for i in range(n)]
l = ray.get(futures)
print(l)
return l
@workflow
def ray_workflow(n: int) -> typing.List[int]:
return ray_task(n=n)
if __name__ == "__main__":
f.remote(x=2)
image
2. rayjob fails
import typing
from flytekit import ImageSpec, Resources, task, workflow
commit_hash = "5a57f299c1ae789c9e2f269302becf13ca3b21e8"
ray_plugin = f"git+<https://github.com/flyteorg/flytekit.git@{commit_hash}#subdirectory=plugins/flytekit-ray>"
custom_image = ImageSpec(
name="ray-flyte-plugin",
registry="localhost:30000",
packages=[ray_plugin],
apt_packages=['git', 'wget', 'vim'],
)
if custom_image.is_container():
import ray
from flytekitplugins.ray import HeadNodeConfig, RayJobConfig, WorkerNodeConfig
@ray.remote
def f(x):
a = 1 // 0
print(f"ray task get {x}")
return x * x
ray_config = RayJobConfig(
head_node_config=HeadNodeConfig(ray_start_params={"log-color": "True"}),
worker_node_config=[WorkerNodeConfig(group_name="ray-group", replicas=1)]
)
@task(
task_config=ray_config,
requests=Resources(mem="4Gi", cpu="1"),
container_image=custom_image,
)
def ray_task(n: int) -> typing.List[int]:
futures = [f.remote(i) for i in range(n)]
l = ray.get(futures)
print(l)
return l
@workflow
def ray_workflow(n: int) -> typing.List[int]:
return ray_task(n=n)
if __name__ == "__main__":
f.remote(x=2)
image
Docs link
TODO: Need to add doc for this backward incompatible changes
flyteorg/flytekit
✅ All checks have passed
2/2 successful checksGitHub
03/16/2024, 12:02 AM1.1.0-rc.0
2. Import from rayv1
instead of rayv1alpha1
, which will be deprecated
3. Replace runtime env with runtime yaml because the former is deprecated already
4. Re-map rayjob status to flyte task status. As suggested by @Yicheng-Lu-llll , only rely on jobdeployment status
How was this patch tested?
Check flyteorg/flytekit#2274
Related PRs
flyteorg/flytekit#2274
Docs link
Needs to add doc somewhere
flyteorg/flyte
✅ All checks have passed
2/2 successful checksGitHub
03/16/2024, 1:30 AMGitHub
03/16/2024, 5:17 AM<https://github.com/flyteorg/flytesnacks/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytesnacks/commit/a2722de42df046a342944ce4f526764c8bce3600|a2722de4>
- Add Jupyter notebook example to FlyteInteractive docs (#1551)
flyteorg/flytesnacksGitHub
03/16/2024, 6:45 AMCreate KinD cluster
and ct install
steps temporarily until we prepare all needed dependencies.
How was this patch tested?
test on commit "7773afb"
https://github.com/lowc1012/flyte/actions/runs/8306349553
Setup process
Screenshots
Check all the applicable boxes
☐ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
Docs link
flyteorg/flyte
Codecov: codecov/project
GitHub Actions: build-and-push-sandbox-bundled-image
✅ 28 other checks have passed
28/30 successful checksGitHub
03/16/2024, 1:10 PMallow="clipboard-write"
attribute to the SVG element
Type
☐ Bug Fix
☑︎ Feature
☐ Plugin
Are all requirements met?
☑︎ Code completed
☐ Smoke tested
☐ Unit tests added
☐ Code documentation added
☐ Any pending items have an associated Issue
Complete description
In order to implement the copy function on the iframe, the iframe needs to add attributes.
For the reasons, please refer to: flyteorg/flytekit#2264 (review)
Tracking Issue
NA
Follow-up issue
flyteorg/flytekit#2264
flyteorg/flyteconsole
GitHub Actions: Build & Push FlyteConsole Image
GitHub Actions: Get Release Tag
GitHub Actions: Generate Release
✅ 8 other checks have passed
8/11 successful checksGitHub
03/18/2024, 7:22 AMGitHub
03/18/2024, 8:01 AM<http://ghcr.io/flyteorg/rawcontainers-julia:v2|ghcr.io/flyteorg/rawcontainers-julia:v2>
, and start the image as container, we will find that julia
command is not available.
image
Reference: https://docs.flyte.org/en/latest/user_guide/customizing_dependencies/raw_containers.html#raw-container
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/18/2024, 8:25 AMtask
, workflow
, and launchplan
x2.5 ~ x4 faster in pyflyte run
by leveraging asyncio
.
What changes were proposed in this pull request?
• #2267: A neat workaround by @pingsutw for sync grpc clients.
How was this patch tested?
Setup process
After introducing localhost network latency to 1000ms
per request, we can simulate IO-intensive operations to cloud services (e.g., object storage, Kubernetes cluster, FlyteAdmin) on your own desktop. This way, we can finally evaluate performance improvements through asyncio
without setup a real cluster outside. Otherwise, the Flyte sandbox cluster residing in macOS is just too fast to measure.
Steps to reproduce network condition:
1. `sudo dnctl pipe 1 config delay 1000`: Adding 1000ms latency to every requests.
2. `sudo pfctl -E`: Enabling the latency setup
Don't forget set latency back toto get your original network speed.0
Screenshot 2024-03-18 at 3 15 17 PMScreenshots • The registration elapsed time reduced from
34.1s
to 16.1s
after introducing asynchronous registration.
*1. Register in the sync
manner: `34.1s`*
1. python -m cProfile -o sync_requests.prof ./flytekit/clis/sdk_in_container/pyflyte.py run --remote ./workflow.py wf
2. snakeviz ./sync_requests.prof
• Screenshot 2024-03-18 at 3 14 05 PM
*2. Register in the async
manner: `16.1s`*
1. python -m cProfile -o async_requests.prof ./flyte kit/clis/sdk_in_container/pyflyte.py run --remote . /workflow.py wf
2. snakeviz ./async_requests.prof
• Screenshot 2024-03-18 at 3 14 13 PM
Check all the applicable boxes
☐ I updated the documentation accordingly.
☐ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
Docs link
flyteorg/flytekit
WIP: Title contains "WIP"
✅ 1 other check has passed
1/2 successful checksGitHub
03/18/2024, 9:12 AMfrom flytekit import task, workflow, ImageSpec, dynamic
new_flytekit = "git+<https://github.com/flyteorg/flytekit@64730c8ffb4e76cb268817e30d31c6d209452e75>"
image_sklearn = ImageSpec(packages=["scikit-learn", new_flytekit], apt_packages=["git"], registry="pingsutw")
image_tensorflow = ImageSpec(base_image=image_sklearn, packages=["tensorflow"], registry="pingsutw")
@task(container_image=image_sklearn)
def t1(a: int) -> int:
return a + 2
@workflow
def wf(a: int = 3):
t1(a=a)
if __name__ == "__main__":
print(f"Running my_wf: {wf(a=50)}")
Screenshots
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
NA
Docs link
NA
flyteorg/flytekit
Codecov: 83.21% (-0.58%) compared to 3f45131
Codecov: 44.44% of diff hit (target 83.79%)
✅ 28 other checks have passed
28/30 successful checksGitHub
03/18/2024, 11:35 AMjupyter
to ClassDecorator
. Having the ClassDecorator
for jupyter
enables to call on get_extra_config
, which can be used to enable dynamic links in Flyte.
What changes were proposed in this pull request?
• putting source for jupyter into ClassDecorator
functions: execute()
, exit_handler()
• added write_example_notebook()
• refactored the use of constants for vs_code
& jupyter
• moved function execute_command()
to utils
How was this patch tested?
• unit tests added
• tested remotely on Flyte Cluster
Setup process
Docker Image being used:
FROM python:3.9-slim-bookworm
WORKDIR /root
ENV PYTHONPATH /root
ENV FLYTE_SDK_RICH_TRACEBACKS 0
RUN apt-get update && apt-get install build-essential -y \
&& useradd -u 1000 flytekit \
&& chown flytekit: /root \
&& chown flytekit: /home \
&& :
RUN pip install flytekit
COPY plugins/flytekit-flyteinteractive plugins/flytekit-flyteinteractive
RUN pip install -e plugins/flytekit-flyteinteractive/
USER flytekit
Worfklow being used:
from flytekit import conditional, dynamic, task, workflow
from flytekit.core.resources import Resources
from flytekitplugins.flyteinteractive import jupyter, vscode
@task
@jupyter
def jupyter_task():
print("forward")
print("backward")
@workflow
def wf():
jupyter_task().with_overrides(limits=Resources(cpu="1", mem="1G"))
if __name__ == "__main__":
print(wf())
Screenshots
Kubectl port forward:
image
Dynamic Links can be configured:
image
⚠️ Jupyter redirects is not working at the moment. This we need to figure out.
image
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
Docs link
flyteorg/flytekit
Codecov: 83.30% (-0.50%) compared to 3f45131
Codecov: 83.06% of diff hit (target 83.79%)
✅ 28 other checks have passed
28/30 successful checksGitHub
03/18/2024, 1:41 PMpython_val.to_json()
to JSONEncoder(python_type).encode(python_val)
2. change expected_python_type.from_json(json_str)
to JSONDecoder(expected_python_type).decode(json_str)
3. change return dataclass_json(dataclasses.make_dataclass(schema_name, attribute_list))
to
return dataclasses.make_dataclass(schema_name, attribute_list)
4. add tests
5. fix mypy errors
6. change type annotations
How was this patch tested?
1. unit tests
2. local and remotely with only dataclass
decorator
3. local and remotely with dataclass inherits from DataClassJSONMixin
. (for backward compatible)
Setup process
python dataclass_example.py
pyflyte run --remote --image localhost:30000/dataclass:0951 dataclass_example.py dataclass_wf --x 10 --y 20
import os
import tempfile
from dataclasses import dataclass
from typing import Tuple, List, Optional
import pandas as pd
from flytekit import task, workflow
from flytekit.types.directory import FlyteDirectory
from flytekit.types.file import FlyteFile
from flytekit.types.structured import StructuredDataset
# from mashumaro.mixins.json import DataClassJSONMixin
@dataclass
class Datum:
x: int
y: str
z: dict[int, int]
w: List[int] = None
@task
def stringify(s: int) -> Datum:
"""
A dataclass return will be treated as a single complex JSON return.
"""
return Datum(x=s, y=str(s), z={s: str(s)}, w=[s,s,s,s])
@task
def add(x: Datum, y: Datum) -> Datum:
"""
Flytekit automatically converts the provided JSON into a data class.
If the structures don't match, it triggers a runtime failure.
"""
x.z.update(y.z)
return Datum(x=x.x + y.x, y=x.y + y.y, z=x.z, w=x.w + y.w)
@dataclass
class FlyteTypes:
dataframe: StructuredDataset
file: FlyteFile
directory: FlyteDirectory
@task
def upload_data() -> FlyteTypes:
"""
Flytekit will upload FlyteFile, FlyteDirectory and StructuredDataset to the blob store,
such as GCP or S3.
"""
# 1. StructuredDataset
df = pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]})
# 2. FlyteDirectory
temp_dir = tempfile.mkdtemp(prefix="flyte-")
df.to_parquet(temp_dir + "/df.parquet")
# 3. FlyteFile
file_path = tempfile.NamedTemporaryFile(delete=False)
file_path.write(b"Hello, World!")
fs = FlyteTypes(
dataframe=StructuredDataset(dataframe=df),
file=FlyteFile(file_path.name),
directory=FlyteDirectory(temp_dir),
)
return fs
@task
def download_data(res: FlyteTypes):
assert pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]}).equals(res.dataframe.open(pd.DataFrame).all())
f = open(res.file, "r")
assert f.read() == "Hello, World!"
assert os.listdir(res.directory) == ["df.parquet"]
@workflow
def dataclass_wf(x: int, y: int) -> Tuple[Datum, FlyteTypes]:
o1 = add(x=stringify(s=x), y=stringify(s=y))
o2 = upload_data()
download_data(res=o2)
return o1, o2
if __name__ == "__main__":
print(dataclass_wf(x=10, y=20))
FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
RUN pip install -U git+<https://github.com/flyteorg/flytekit.git@c1d7075428c6352abc11df825d2e3406f70f91fd>
RUN pip install pandas -U
Screenshots
local execution (with only dataclass decorator)
image
local execution (with DataClassJSONMixin
)
image
remote execution (with only dataclass decorator)
image image
remote execution (with DataClassJSONMixin
)
image <https://private-user-images.githubusercontent.com/76461262/313724495-95cebaf7-d8cc-4eda-bd…
flyteorg/flytekit
GitHub Actions: build (macos-latest, 3.12)
GitHub Actions: integration (ubuntu-latest, 3.12)
GitHub Actions: build (windows-latest, 3.12)
GitHub Actions: build-with-extras (windows-latest, 3.12)
GitHub Actions: build (ubuntu-latest, 3.12)
GitHub Actions: build-with-pandas (ubuntu-latest, 3.12, pandas>=2.0.0)
GitHub Actions: build-with-extras (ubuntu-latest, 3.12)
GitHub Actions: build-with-pandas (ubuntu-latest, 3.12, pandas<2.0.0)
GitHub Actions: test-serialization (ubuntu-latest, 3.12)
GitHub Actions: Monodocs Build
GitHub Actions: lint
✅ 4 other checks have passed
4/15 successful checksGitHub
03/18/2024, 1:45 PMGitHub
03/18/2024, 1:45 PMGitHub
03/18/2024, 4:00 PMGitHub
03/18/2024, 4:00 PMGitHub
03/18/2024, 5:10 PMfrom flytekit import task, workflow
@task(cache=True, cache_version="v1", cache_ignore_input_vars=["a"])
def add(a: int, b: int) -> int:
return a + b
@workflow
def add_wf(a: int, b: int) -> int:
return add(a=a, b=b)
if __name__ == "__main__":
assert add_wf(a=10, b=5) == 15
assert add_wf(a=20, b=5) == 15 # since a is ignored, this line will hit cache of a=10, b=5
assert add_wf(a=20, b=8) == 28
Related PRs
flyteorg/flyte#4618
flyteorg/flytekit
Codecov: 83.32% (-0.68%) compared to 96f2b2a
✅ 29 other checks have passed
29/30 successful checksGitHub
03/18/2024, 5:23 PMGitHub
03/18/2024, 5:52 PM<https://github.com/flyteorg/flytekit/tree/master|master>
by pingsutw
<https://github.com/flyteorg/flytekit/commit/c8ac2764f2de8d7e98e56bcf5c7a04ab07e617ca|c8ac2764>
- cache ignore portion (#2275)
flyteorg/flytekitGitHub
03/18/2024, 6:02 PM<https://github.com/flyteorg/flyte/tree/master|master>
by katrogan
<https://github.com/flyteorg/flyte/commit/9818423079d265bfc44b728f8d10b2fb0b2944db|98184230>
- Add org as an optional request param to dataproxy CreateUploadLocation (#5060)
flyteorg/flyteGitHub
03/18/2024, 6:59 PMpyflyte run --remote flyte-example/test2.py spark_wf
Setup process
import random
from click.testing import CliRunner
from flytekit.clis.sdk_in_container import pyflyte
from flytekitplugins.spark import Spark
import flytekit
from flytekit import ImageSpec, task, Resources, workflow
from operator import add
custom_image = ImageSpec(
name="flytekit",
registry="pingsutw", # your docker registry
)
def f(_):
x = random.random() * 2 - 1
y = random.random() * 2 - 1
return 1 if x**2 + y**2 <= 1 else 0
@task(
task_config=Spark(
spark_conf={
"spark.driver.memory": "1000M",
"spark.executor.memory": "1000M",
"spark.executor.cores": "1",
"spark.executor.instances": "2",
"spark.driver.cores": "1",
"spark.jars": "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar",
},
),
limits=Resources(mem="2000M"),
container_image=custom_image,
)
def hello_spark(partitions: int) -> float:
print("Starting Spark with Partitions: {}".format(partitions))
n = 1 * partitions
sess = flytekit.current_context().spark_session
count = (
sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
)
pi_val = 4.0 * count / n
return pi_val
@workflow
def spark_wf(partitions: int = 10) -> float:
return hello_spark(partitions=partitions)
if __name__ == '__main__':
runner = CliRunner()
result = runner.invoke(pyflyte.main,
["register",
"--non-fast",
"--version",
"v3",
"/Users/kevin/git/flytekit/flyte-example/test2.py",
])
print(result.output)
Screenshots
image
☑︎ I updated the documentation accordingly.
☑︎ All new and existing tests passed.
☑︎ All commits are signed-off.
Related PRs
NA
Docs link
NA
flyteorg/flytekit
Codecov: 83.28% (-0.51%) compared to 3f45131
✅ 29 other checks have passed
29/30 successful checksGitHub
03/18/2024, 7:10 PMGitHub
03/18/2024, 7:40 PMGitHub
03/18/2024, 9:15 PMGitHub
03/18/2024, 9:35 PM<https://github.com/flyteorg/flyte/tree/master|master>
by eapolinario
<https://github.com/flyteorg/flyte/commit/eb2410780edf5f525429f4d83d8bdd2397f08cc8|eb241078>
- Add k8s env froms (#4969)
flyteorg/flyteGitHub
03/18/2024, 9:57 PMGitHub
03/18/2024, 10:44 PM@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
...
According to the docs, one of the inputs to cache key calculation is the task signature, but in the case of this example, it'd be great if we could ignore log_level
as part of the cache key calculation.
Goal: What should the final outcome look like, ideally?
We should be able to do something along the lines of:
@task(cache=True, cache_version="v1", ignore_input_vars=["log_level"])
def t(log_level: int, a: str) -> str:
...
This would essentially skip some of the parameters for cache key calculation purposes.
Describe alternatives you've considered
We have the ability to override the hashing mechanism used to translate python types into Flyte types, as described in https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects.
One could use this idea and provide constant hashes for the arguments they want to ignore, for example:
def constant_function(x: int) -> str:
return "const"
@task
def t_produce_annotated_literals() -> Annotated[int, HashMethod(constant_function)]:
log_level = ...
return log_level
@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
...
@workflow
def wf() -> str:
log_level = t_produce_annotated_literals()
return t(log_level=log_level, a="some string")
Propose: Link/Inline OR Additional context
Expose ignore_input_vars
in the @task
decorator and ensure the new interface is used during cache key calculation in both local and remote executions.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
03/19/2024, 1:33 AMfrom dev_python_dependency_deck import py_deps_deck_wf
from flytekit import LaunchPlan, CronSchedule
py_deps_deck_lp = LaunchPlan.get_or_create(
py_deps_deck_wf,
name="py_deps_deck_lp",
schedule=CronSchedule(schedule="06 00 * * *"),
)
and I execute cmd: FLYTE_SDK_LOGGING_LEVEL=20 pyflyte register --non-fast -p flytesnacks --domain development dev_register_lp.py --version t1
then I found that the result was...
{"asctime": "2024-03-19 090219,831", "name": "flytekit", "levelname": "INFO", "message": "Registering Sensor for task type: sensor_v0"}
Running pyflyte register from /Users/al02027975/Documents/jasonlai1218/flytekit with images ImageConfig(default_image=Image(name='default', fqn='cr.flyte.org/flyteorg/flytekit', tag='py3.11-latest'), images=[Image(name='default', fqn='cr.flyte.org/flyteorg/flytekit', tag='py3.11-latest')]) and image destination folder /root on 1 package(s) ('/Users/al02027975/Documents/jasonlai1218/flytekit/dev_register_lp.py',)
Registering against localhost:30080
Detected Root /Users/al02027975/Documents/jasonlai1218/flytekit, using this to create deployable package...
Loading packages ['dev_register_lp'] under source root /Users/al02027975/Documents/jasonlai1218/flytekit
Image jasonlai1218/flytekit:Lfx23vy2B9huZasN473ypQ not found. building...
Run command: envd build --path /var/folders/r7/cgsr7w_n4v148k8qbryjc6ym0000gp/T/flyte75ltk3sy/control_plane_metadata/local_flytekit/4b55674ab034d8c89e440cef577d0ca1 --platform linux/amd64 --output type=image,name=jasonlai1218/flytekit:Lfx23vy2B9huZasN473ypQ,push=true
#1 local://build-context
#1 DONE 0.0s
#2 [internal] setting pip cache mount permissions
#2 DONE 0.0s
#3 docker-image://cr.flyte.org/flyteorg/flytekit:py3.11-latest
...
...
...
#14 pushing layers
#14 pushing layers 16.9s done
#14 pushing manifest for docker.io/jasonlai1218/flytekit:Lfx23vy2B9huZasN473ypQ@sha256:54ef656aaf12784fdcaacd600079e0c024271e39ab81b4f925d35e785c0e1164
#14 pushing manifest for docker.io/jasonlai1218/flytekit:Lfx23vy2B9huZasN473ypQ@sha256:54ef656aaf12784fdcaacd600079e0c024271e39ab81b4f925d35e785c0e1164 0.7s done
#14 DONE 23.3s
Successfully serialized 4 flyte objects
[✔️] Registration dev_python_dependency_deck.python_dependency_deck type TASK successful with version t1
[✔️] Registration dev_python_dependency_deck.py_deps_deck_wf type WORKFLOW successful with version t1
[✔️] Registration dev_python_dependency_deck.py_deps_deck_wf type LAUNCH_PLAN successful with version t1
[✔️] Registration py_deps_deck_lp type LAUNCH_PLAN successful with version t1
Successfully registered 4 entitiesI don't know why there are two
LAUNCH_PLAN
Expected behavior
I think only one is correct
[✔️] Registration py_deps_deck_lp type LAUNCH_PLAN successful with version t1Additional context to reproduce 1. write a workflow and task
import flytekit
from flytekit import task, workflow, ImageSpec
from flytekit.deck.deck import PythonDependencyDeck
flytekit_version = "git+<https://github.com/jasonlai1218/flytekit.git@add-py-deps-deck#commit=341a7dab611e31023ae82628f5aad5792b7c5d51>"
deck_plugin_version = "git+<https://github.com/jasonlai1218/flytekit.git@add-py-deps-deck#commit=341a7dab611e31023ae82628f5aad5792b7c5d51#subdirectory=plugins/flytekit-deck-standard>"
image_spec = ImageSpec(
packages=[flytekit_version, deck_plugin_version],
apt_packages=["git"],
registry="jasonlai1218",
env={"FLYTE_SDK_LOGGING_LEVEL": "20"},
)
@task(container_image=image_spec, enable_deck=True)
def python_dependency_deck() -> None:
ctx = flytekit.FlyteContextManager.current_context()
ctx.add_deck(PythonDependencyDeck())
@workflow
def py_deps_deck_wf() -> None:
python_dependency_deck()
if __name__ == "__main__":
py_deps_deck_wf()
2. write a launch plan
from dev_python_dependency_deck import py_deps_deck_wf
from flytekit import LaunchPlan, CronSchedule
py_deps_deck_lp = LaunchPlan.get_or_create(
py_deps_deck_wf,
name="py_deps_deck_lp",
schedule=CronSchedule(schedule="06 00 * * *"),
)
3. execute cmd: FLYTE_SDK_LOGGING_LEVEL=20 pyflyte register --non-fast -p flytesnacks --domain development dev_register_lp.py --version t1
Screenshots
Screenshot 2024-03-19 at 9 30 10 AM
Screenshot 2024-03-19 at 9 29 52 AM
Screenshot 2024-03-19 at 9 11 46 AM
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
…
flyteorg/flyte