thousands-dawn-39963
02/14/2023, 9:44 PMcontainers with unready status: [f7592492fe4da4f14b2f-n0-0]|Failed to apply default image tag "<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image|cr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image>": couldn't parse image reference "<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image|cr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image>": invalid reference format
broad-monitor-993
02/14/2023, 9:53 PM<http://ghcr.flyte.org/|ghcr.flyte.org/>…
not <http://cr.flyte.org|cr.flyte.org>
thousands-dawn-39963
02/14/2023, 9:54 PMthousands-dawn-39963
02/14/2023, 10:05 PMcontainers with unready status: [fba6b54cf32154cd6a01-n0-0]|Failed to apply default image tag "<http://ghcr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image|ghcr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image>": couldn't parse image reference "<http://ghcr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image|ghcr.flyte.org/flyteorg/flytekit:py3.10-1.3.2:image>": invalid reference format
future-monitor-58430
02/14/2023, 10:12 PMfqn="<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>",
tag="py3.10-1.3.2",
thousands-dawn-39963
02/14/2023, 10:28 PMthousands-dawn-39963
02/14/2023, 11:22 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f5743f26504394989994-n0-0] terminated with exit code (1). Reason [Error]. Message:
ages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
_execute_task(
File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
_task_def = resolver_obj.load_task(loader_args=resolver_args)
File "/usr/local/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 280, in load_task
task_module = importlib.import_module(task_module)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'app'
.
high-accountant-32689
02/14/2023, 11:53 PMthousands-dawn-39963
02/15/2023, 12:27 AMthousands-dawn-39963
02/15/2023, 12:28 AMself.remote.register_task(
self.sum_test_2,
version=VERSION,
serialization_settings=serialization_settings,
)
thousands-dawn-39963
02/15/2023, 12:28 AMself.remote.register_workflow(
process_data_wf_3,
version=VERSION,
serialization_settings=inner_serialization_settings,
)
thousands-dawn-39963
02/15/2023, 12:29 AMlaunch_plan = LaunchPlan.get_or_create(
process_data_wf_3,
name="wf_launchplan_3",
schedule=FixedRate(duration=timedelta(minutes=1)),
)
thousands-dawn-39963
02/15/2023, 12:30 AMthousands-dawn-39963
02/15/2023, 12:33 AMfuture-monitor-58430
02/15/2023, 1:23 AMSerializationSettings
thousands-dawn-39963
02/15/2023, 4:34 AMthousands-dawn-39963
02/15/2023, 5:31 AMserialization_settings = SerializationSettings(
project="flytesnacks",
domain=self.settings.flyte_domain,
env=None,
image_config=ImageConfig(
default_image=Image(
name="custom_container_task",
fqn="<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>",
tag="py3.10-1.3.2",
)
),
fast_serialization_settings=FastSerializationSettings(
enabled=True,
destination_dir='/Users/rezwanabir/Desktop'
)
)
thousands-dawn-39963
02/15/2023, 5:32 AM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ff6805959e365479e99a-n0-0] terminated with exit code (1). Reason [Error]. Message:
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/usr/local/lib/python3.10/site-packages/flytekit/tools/fast_registration.py", line 107, in download_distribution
raise ValueError("Destination path is required to download distribution and it should be a directory")
ValueError: Destination path is required to download distribution and it should be a directory
.
thousands-dawn-39963
02/15/2023, 5:38 AMdestination_dir='.'
thousands-dawn-39963
02/15/2023, 5:38 AMneed fast serialization or else will fail
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f7ee0d9e0f0204756b5d-n0-0] terminated with exit code (1). Reason [Error]. Message:
oe
File "/usr/local/lib/python3.10/site-packages/flytekitplugins/fsspec/persist.py", line 99, in get
return fs.get(from_path, to_path, recursive=recursive)
File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 891, in get
self.get_file(rpath, lpath, **kwargs)
File "/usr/local/lib/python3.10/site-packages/fsspec/implementations/local.py", line 135, in get_file
return self.cp_file(path1, path2, **kwargs)
File "/usr/local/lib/python3.10/site-packages/fsspec/implementations/local.py", line 128, in cp_file
raise FileNotFoundError(path1)
FileNotFoundError: /root/{{ .remote_package_path }}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/usr/local/lib/python3.10/site-packages/flytekit/tools/fast_registration.py", line 111, in download_distribution
FlyteContextManager.current_context().file_access.get_data(additional_distribution, os.path.join(destination, ""))
File "/usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 456, in get_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from {{ .remote_package_path }} to ./ (recursive=False).
Original exception: /root/{{ .remote_package_path }}
.
future-monitor-58430
02/15/2023, 5:44 AMfuture-monitor-58430
02/15/2023, 5:45 AMthousands-dawn-39963
02/15/2023, 6:04 AMbroad-monitor-993
02/15/2023, 2:37 PMfast_serialization
config Flyte will automatically zip up your local files, dump it in the configured blob store, then overlay your code on top of the specified docker image. Not sure if that answers your question.thousands-dawn-39963
02/15/2023, 2:41 PMthousands-dawn-39963
02/15/2023, 2:42 PMbroad-monitor-993
02/15/2023, 3:15 PMthousands-dawn-39963
02/15/2023, 3:24 PMbroad-monitor-993
02/15/2023, 4:17 PMFlyteRemote
to do it? @high-accountant-32689 @thankful-minister-83577 is there a way to update the launchplan schedule in-place? Or do folks have to deactivate the old version, register a new version, and active the new version?thousands-dawn-39963
02/15/2023, 4:22 PMthousands-dawn-39963
02/15/2023, 4:24 PMbroad-monitor-993
02/15/2023, 4:24 PMbroad-monitor-993
02/15/2023, 4:25 PMthousands-dawn-39963
02/15/2023, 4:26 PMthousands-dawn-39963
02/15/2023, 4:26 PMbroad-monitor-993
02/15/2023, 4:34 PMfrom workflows import wf
remote = FlyteRemote(...)
users, cron_schedules = get_cron_schedules_per_user()
for user, cron_schedule in zip(users, cron_schedules):
# by default, this gets the latest version, but you can pass in
# a `version="..."` argument
old_flyte_lp = remote.fetch_launch_plan(name=f"launchplan_{user}")
# deactivate old launchplan
remote.client.update_launch_plan(old_flyte_lp.id, "INACTIVE")
# create new launchplan
new_lp = LaunchPlan.get_or_create(
wf,
name=f"launchplan_{user}",
schedule=CronSchedule(schedule=cron_schedule, ...),
)
new_flyte_lp = remote.register_launch_plan(new_lp, version="new_version")
remote.client.update_launch_plan(new_flyte_lp, "ACTIVE")
broad-monitor-993
02/15/2023, 4:36 PM@dynamic
workflow (see docs), but you’ll need to configure it such that the docker image running the dynamic workflow pod can access your Flyte cluster.thousands-dawn-39963
02/15/2023, 4:39 PMbroad-monitor-993
02/15/2023, 5:35 PMFlyteRemote
can handle this, you need to provide FastSerializationSettings to your SerializationSettings config. We’ll need to document this, but here’s an example:
from flytekit.tools import fast_registration, repo
detected_root = repo.find_common_root(["."])
zip_file = fast_registration.fast_package(detected_root, output_dir=None, deref_symlinks=False)
# Upload zip file to Admin using FlyteRemote.
_, native_url = remote.fast_package(detected_root)
fast_serialization_settings = FastSerializationSettings(
enabled=True,
distribution_location=native_url,
)
serialization_settings = SerializationSettings(
...
fast_serialization_settings=fast_serialization_settings,
)
# now you can register the launch plan as in the code snippet about
remote.register_launch_plan(lp, serialization_settings, version)
Would you mind opening up a [flyte-docs] issue for documentating this better? 👇user
02/15/2023, 5:35 PMthousands-dawn-39963
02/15/2023, 5:49 PMlaunch_plan = LaunchPlan.get_or_create(
process_data_wf_3,
name="wf_launchplan_3",
schedule=FixedRate(duration=timedelta(minutes=1)),
default_inputs={"x": 4, "y": 5},
)
execution = self.remote.execute(
launch_plan,
inputs={},
version="kbPtdOiwwodI6FUmBip_wA==",
project="flytesnacks",
domain=self.settings.flyte_domain,
)
thousands-dawn-39963
02/15/2023, 5:50 PMbroad-monitor-993
02/15/2023, 5:52 PMremote.client.update_launch_plan(new_flyte_lp, "ACTIVE")
broad-monitor-993
02/15/2023, 5:53 PMremote.execute
will just run the launch plan outside of the schedule.
Activating the schedule will make it run on the configured schedule. Now if you want to update the launch plan schedule, you’ll have to do the steps I mentioned before to deactivate the old launchplan version, register the new one, and activate the new one. Does that make sense?thousands-dawn-39963
02/15/2023, 9:52 PMbroad-monitor-993
02/15/2023, 9:53 PMACTIVE
ate the new version of the launchplan and that will automatically deactivate the old onesthousands-dawn-39963
02/15/2023, 9:54 PMthousands-dawn-39963
02/15/2023, 11:07 PMCACHED OUT PUT.... {'_name': 'launch_plan_4a6bdc85-5652-4791-9347-bed36a86b4ee', '_workflow': WorkflowBase - app.flyte.tasks.process_data_wf_3 && Inputs (2): {'x': <class 'int'>, 'y': <class 'int'>} && Outputs (0): {} && Output bindings: [] && , '_parameters': parameters {
key: "x"
value {
var {
type {
simple: INTEGER
}
description: "x"
}
default {
scalar {
primitive {
integer: 4
}
}
}
}
}
parameters {
key: "y"
value {
var {
type {
simple: INTEGER
}
description: "y"
}
default {
scalar {
primitive {
integer: 5
}
}
}
}
}
, '_fixed_inputs': , '_saved_inputs': {'x': 4, 'y': 5}, '_schedule': cron_schedule {
schedule: "*/1 * * * *"
}
, '_notifications': [], '_labels': None, '_annotations': None, '_raw_output_data_config': None, '_max_parallelism': None, '_security_context': None}
ERRORR>>>>>>>>>>>>
Error at execute_custom_dashboard_workflow..... The cached values aren't the same as the current call arguments
thousands-dawn-39963
02/15/2023, 11:07 PMtry:
cron_expression = request.get("cron")
print("cron expression", cron_expression)
detected_root = repo.find_common_root(["."])
_, fast_url = self.fast_package(detected_root, deref_symlinks=False)
fast_serialization_settings = FastSerializationSettings(
enabled=True, distribution_location=fast_url, destination_dir="/root"
)
serialization_settings = SerializationSettings(
project="flytesnacks",
domain=self.settings.flyte_domain,
env=None,
image_config=ImageConfig(
default_image=Image(
name="custom_container_task",
fqn="<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>",
tag="py3.10-1.3.2",
)
),
fast_serialization_settings=fast_serialization_settings,
)
launch_plan_name = f"launch_plan_{str(report_id)}"
self.register_task(serialization_settings)
self.register_workflow(serialization_settings)
# self.disable_old_launch_plan(launch_plan_name)
launch_plan = LaunchPlan.get_or_create(
process_data_wf_3,
name=launch_plan_name,
schedule=CronSchedule(schedule=cron_expression),
default_inputs={"x": 4, "y": 5},
)
flyte_lp_id = self.register_launch_plan(launch_plan)
self.remote.client.update_launch_plan(flyte_lp_id.id, "ACTIVE")
print("Success!!")
except Exception as e:
print("Error at execute_custom_dashboard_workflow.....", e)
return launch_plan
thousands-dawn-39963
02/15/2023, 11:15 PMbroad-monitor-993
02/15/2023, 11:23 PMthousands-dawn-39963
02/15/2023, 11:25 PMthousands-dawn-39963
02/15/2023, 11:26 PMthousands-dawn-39963
02/15/2023, 11:27 PMtall-lock-23197