rough-rose-81585
03/15/2022, 4:06 PMPriorityClass: system-cluster-critical
attribute which requires an associated k8s quota
that was not generated by my helm config. Manually creating one allowed Flytepropeller pods to start.
Was this documented anywhere? What do I need to add to my helm values file to generate this quota automatically?great-school-54368
03/15/2022, 4:10 PMfreezing-airport-6809
freezing-airport-6809
great-school-54368
03/15/2022, 4:22 PMgreat-school-54368
03/15/2022, 4:24 PMapiVersion: <http://scheduling.k8s.io/v1|scheduling.k8s.io/v1>
description: Used for system critical pods that must run in the cluster, but can be
moved to another node if necessary.
kind: PriorityClass
metadata:
name: system-cluster-critical
preemptionPolicy: PreemptLowerPriority
value: 2000000000
---
apiVersion: <http://scheduling.k8s.io/v1|scheduling.k8s.io/v1>
description: Used for system critical pods that must not be moved from their current
node.
kind: PriorityClass
metadata:
name: system-node-critical
preemptionPolicy: PreemptLowerPriority
value: 2000001000
high-park-82026
rough-rose-81585
03/15/2022, 4:29 PMhigh-park-82026
Note: Kubernetes already ships with two PriorityClasses:I’m trying to figure out if GKE explicitly removes these…andsystem-cluster-critical
. These are common classes and are used to ensure that critical components are always scheduled first.system-node-critical
rough-rose-81585
03/15/2022, 4:38 PMrough-rose-81585
03/15/2022, 4:38 PMhigh-park-82026
rough-rose-81585
03/15/2022, 4:41 PMapiVersion: v1
kind: ResourceQuota
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"v1","kind":"ResourceQuota","metadata":{"annotations":{},"name":"sys-cluster-critical-quota","namespace":"flyte"},"spec":{"hard":{"cpu":"1000","memory":"2000Gi","pods":"100"},"scopeSelector":{"matchExpressions":[{"operator":"In","scopeName":"PriorityClass","values":["system-cluster-critical"]}]}}}
creationTimestamp: "2022-03-15T02:02:40Z"
name: sys-cluster-critical-quota
namespace: flyte
resourceVersion: "151298561"
uid: e0d211f6-8523-4032-8f5e-04b0a449b855
spec:
hard:
cpu: 1k
memory: 2000Gi
pods: "100"
scopeSelector:
matchExpressions:
- operator: In
scopeName: PriorityClass
values:
- system-cluster-critical
status:
hard:
cpu: 1k
memory: 2000Gi
pods: "100"
used:
cpu: 500m
memory: 2Gi
pods: "1"
rough-rose-81585
03/15/2022, 4:41 PMhigh-park-82026
high-park-82026
rough-rose-81585
03/15/2022, 4:43 PMrough-rose-81585
03/15/2022, 4:43 PMrough-rose-81585
03/15/2022, 4:44 PMhigh-park-82026
high-park-82026
rough-rose-81585
03/15/2022, 4:51 PMhigh-park-82026
high-park-82026
rough-rose-81585
03/15/2022, 5:35 PM[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flytekit/common/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/base_task.py", line 464, in dispatch_execute
native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/type_engine.py", line 600, in literal_map_to_kwargs
return {k: TypeEngine.to_python_value(ctx, lm.literals[k], v) for k, v in python_types.items()}
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/type_engine.py", line 600, in <dictcomp>
return {k: TypeEngine.to_python_value(ctx, lm.literals[k], v) for k, v in python_types.items()}
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/type_engine.py", line 576, in to_python_value
return transformer.to_python_value(ctx, lv, expected_python_type)
File "/usr/local/lib/python3.8/dist-packages/flytekit/core/type_engine.py", line 406, in to_python_value
dc = cast(DataClassJsonMixin, expected_python_type).from_json(_json_format.MessageToJson(lv.scalar.generic))
File "/usr/local/lib/python3.8/dist-packages/dataclasses_json/api.py", line 75, in from_json
return cls.from_dict(kvs, infer_missing=infer_missing)
File "/usr/local/lib/python3.8/dist-packages/dataclasses_json/api.py", line 82, in from_dict
return _decode_dataclass(cls, kvs, infer_missing)
File "/usr/local/lib/python3.8/dist-packages/dataclasses_json/core.py", line 197, in _decode_dataclass
value = _decode_dataclass(field_type, field_value,
File "/usr/local/lib/python3.8/dist-packages/dataclasses_json/core.py", line 208, in _decode_dataclass
return cls(**init_kwargs)
Message:
__call__() missing 1 required positional argument: 'value'
SYSTEM ERROR! Contact platform administrators.
rough-rose-81585
03/15/2022, 5:36 PMlevel1b_products = level1b_dynamic(
o2_level0_files=o2_level0_files,
o2_dark_file=o2_dark_file,
ch4_level0_files=ch4_level0_files,
ch4_dark_file=ch4_dark_file,
ac_file=ac_file,
params=level1b_params,
config=level1b_config,
output_uri=level1b_output_dir,
)
level2_products = level2_dynamic(
level1b_products=level1b_products,
apriori_config=level2_apriori_config,
proxy_config=level2_proxy_config,
proxy_params=level2_proxy_params,
level2_output_uri=level2_output_dir,
)
dynamic task signature
@dynamic(container_image=container_image("LEVEL2_FLYTE_IMAGE"), cache=True, cache_version="1.5.0")
def level2_dynamic(
level1b_products: List[Level1bTaskOutput],
apriori_config: Level2AprioriConfiguration,
proxy_config: Level2ProxyConfiguration,
proxy_params: Level2ProxyParameters,
level2_output_uri: str,
) -> List[FlyteDirectory]:
datacalss definition:
@dataclass_json
@dataclass
class Level1bTaskOutput:
# TODO nlofaso - it seems like a custom dataclass with FlyteFile's can't use type Union[FlyteFile, None]
# initial testing seemed like it just returned the file string. It didn't actually get saved as flytefile
# more testing needed to confirm
# A solution might be to just return only the o2 and ch4 of the desired
# resolution but then you'd lose the other product info
o2_1x1: FlyteFile = empty_flytefile()
o2_5x1: FlyteFile = empty_flytefile()
o2_geoakaze: FlyteFile = empty_flytefile()
ch4_1x1: FlyteFile = empty_flytefile()
ch4_5x1: FlyteFile = empty_flytefile()
ch4_geoakaze: FlyteFile = empty_flytefile()
high-park-82026
high-park-82026
rough-rose-81585
03/15/2022, 6:04 PMrough-rose-81585
03/15/2022, 6:25 PMrough-rose-81585
03/15/2022, 6:35 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
empty_flytefile
?thankful-minister-83577
it seems like a custom dataclass with FlyteFile’s can’t use type Union[FlyteFile, None]should we make a separate ticket for it?
rough-rose-81585
03/15/2022, 7:53 PMdef empty_flytefile():
def noop():
pass
return FlyteFile(
path="<gs://fake/path/to/nowhere>", downloader=noop, remote_path="<gs://fake/path/to/nowhere>"
)
rough-rose-81585
03/15/2022, 7:54 PMUnion[FlyteFile, None]
feature is currently being worked on. I don’t have the ticket number. empty_flytefile()
is my workaround to make those fields optionalthankful-minister-83577
thankful-minister-83577
rough-rose-81585
03/15/2022, 7:59 PMrough-rose-81585
03/15/2022, 7:59 PMthankful-minister-83577
thankful-minister-83577
high-accountant-32689
03/15/2022, 8:09 PMlevel1b_dynamic
also cached by any chance?
Also, can you double-check the versions of flytekit
and dataclass-json
of the failing run?rough-rose-81585
03/16/2022, 1:42 AMflytekit==0.31.0
, dataclasses-json==0.5.6
high-accountant-32689
03/16/2022, 6:41 AMrough-rose-81585
03/16/2022, 3:24 PM@dataclass_json
decorator by mistake and caused the error.
Small test case to reproduce
from dataclasses import dataclass
from dataclasses_json import dataclass_json
from enum import Enum
from flytekit import task, workflow
# ERROR: having these two annotations causes
# TypeError: __call__() missing 1 required positional argument: 'value'
@dataclass_json
@dataclass
class Aggregation(Enum):
GROUP_1x1 = "1x1"
GROUP_5x1 = "5x1"
@dataclass_json
@dataclass
class Nested:
agg: Aggregation
@task
def tsk(nested: Nested):
if nested.agg == Aggregation.GROUP_1x1:
print("1x1")
elif nested.agg == Aggregation.GROUP_5x1:
print("5x1")
else:
raise ValueError("bad aggregation value")
@workflow
def wf():
a = tsk(nested=Nested(agg=Aggregation.GROUP_1x1))
if __name__ == "__main__":
wf()
rough-rose-81585
03/16/2022, 3:24 PMhigh-accountant-32689
03/16/2022, 3:28 PMthankful-minister-83577