Thusal Ranawaka10/15/2022, 10:54 AM
Ryan Nazareth10/18/2022, 1:03 PM
is the parent class for
(see the note on https://www.tensorflow.org/guide/intro_to_modules page). So we need a type transformer for
. But we have two PRs — https://github.com/flyteorg/flytekit/pull/1242 to fix https://github.com/flyteorg/flyte/issues/2759 and https://github.com/flyteorg/flytekit/pull/1241 to fix https://github.com/flyteorg/flyte/issues/2570 that have the same functionality — a type transformer subclassing
. We’ll need to tell one of them to modify the code. So which PR shall we retain?
Ryan Nazareth10/26/2022, 3:39 AM
environment variable values which were printed out after the cluster was setup. Do i need to login to the kubernetes ui and/or configure anything else ?
Sriniketh J10/26/2022, 5:54 AM
needs to be replaced with
in FlyteFile even if it isn’t throwing any mypy errors?
cryptic10/27/2022, 5:30 AM
Ryan Nazareth10/27/2022, 10:50 PM
) as input:
Pod failed. No message received from kubernetes.
`tar: Removing leading
[fb8f63b27ec284bbfbc8-n3-0] terminated with exit code (137). Reason [Error]. Message:
See screenshot below. The error seems like its trying to extract something from tar archive but Im not doing this anywhere in my task logic ? Also, think someone else has reported this error as well https://flyte-org.slack.com/archives/CP2HDHKE1/p1664441568396559?thread_ts=1664433734.963769&cid=CP2HDHKE1
/' from member names
cryptic10/28/2022, 6:40 AM
cryptic10/28/2022, 10:25 AM
? I'm sorry but I'm not quite sure about it (https://github.com/flyteorg/flytekit/actions/runs/3326738618/jobs/5511140308#step:6:360)
cryptic10/28/2022, 5:34 PM
Ryan Nazareth10/28/2022, 9:32 PM
with multiple literal types? If so, how should the
method be defined since
can only accept a single
? See https://github.com/flyteorg/flytekit/pull/1269/files PR for reference.
Ryan Nazareth11/09/2022, 10:53 AM
cryptic11/12/2022, 5:06 AM
Ryan Nazareth11/13/2022, 2:48 AM
Ryan Nazareth11/23/2022, 2:28 PM
, which has tasks to generate TFRecordFile and TFRecordDirectory from tf.train.Example outputs and subsequently deserialise back to
if required. The bit thats causing an error is
which, seems to not be recognising the Annotated TFRecordDatasetV2 type although it should be the expected type in the transformers i have registered here flytekit/extras/tensorflow/record.py. I presume this is why it is defaulting to FlytePickle as in the traceback (this is only my assumption and based on some other logic i have seen elsewhere in the codebase). Also, my other test
@task def consume(dataset: Annotated[TFRecordDatasetV2, TFRecordDatasetConfig(name="testing")]):
which is testing the
logic works fine so i feel something else is an issue which I am finding difficult to spot at the moment.
expected_python_type = typing.Annotated[flytekit.types.pickle.pickle.FlytePickle.__class_getitem__.<locals>._SpecificFormatClass, TFRecordDatasetConfig(compression_type=None, buffer_size=None, num_parallel_reads=None, name='testing')] def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[T]) -> T: uri = lv.scalar.blob.uri # Deserialize the pickle, and return data in the pickle, # and download pickle file to local first if file is not in the local file systems. if ctx.file_access.is_remote(uri): local_path = ctx.file_access.get_random_local_path() ctx.file_access.get_data(uri, local_path, False) uri = local_path with open(uri, "rb") as infile: > data = cloudpickle.load(infile) E _pickle.UnpicklingError: invalid load key, '5'.
Ryan Nazareth11/29/2022, 10:00 PM