Thusal Ranawaka
10/15/2022, 10:54 AMRyan Nazareth
10/18/2022, 1:03 PMSamhita Alla
Samhita Alla
Aditya Bisht
10/24/2022, 5:49 PMSamhita Alla
tf.Module
is the parent class for tf.keras.layers.Layer
and tf.keras.Model
(see the note on https://www.tensorflow.org/guide/intro_to_modules page). So we need a type transformer for tf.Module
. But we have two PRs — https://github.com/flyteorg/flytekit/pull/1242 to fix https://github.com/flyteorg/flyte/issues/2759 and https://github.com/flyteorg/flytekit/pull/1241 to fix https://github.com/flyteorg/flyte/issues/2570 that have the same functionality — a type transformer subclassing tf.keras.Model
. We’ll need to tell one of them to modify the code. So which PR shall we retain?Samhita Alla
Samhita Alla
Ryan Nazareth
10/26/2022, 3:39 AMKUBECONFIG
and FLYTECTL_CONFIG
environment variable values which were printed out after the cluster was setup.
Do i need to login to the kubernetes ui and/or configure anything else ?Sriniketh J
10/26/2022, 5:54 AMSamhita Alla
typing.TypeVar
needs to be replaced with typing.Annotated
in FlyteFile even if it isn’t throwing any mypy errors?cryptic
10/27/2022, 5:30 AMSamhita Alla
Samhita Alla
Samhita Alla
Ryan Nazareth
10/27/2022, 10:50 PM.model
) as input:
Pod failed. No message received from kubernetes.
[fb8f63b27ec284bbfbc8-n3-0] terminated with exit code (137). Reason [Error]. Message:
`tar: Removing leading /' from member names
See screenshot below. The error seems like its trying to extract something from tar archive but Im not doing this anywhere in my task logic ?
Also, think someone else has reported this error as well https://flyte-org.slack.com/archives/CP2HDHKE1/p1664441568396559?thread_ts=1664433734.963769&cid=CP2HDHKE1Samhita Alla
Samhita Alla
cryptic
10/28/2022, 6:40 AMcryptic
10/28/2022, 10:25 AMmake lint
? I'm sorry but I'm not quite sure about it (https://github.com/flyteorg/flytekit/actions/runs/3326738618/jobs/5511140308#step:6:360)cryptic
10/28/2022, 5:34 PMRyan Nazareth
10/28/2022, 9:32 PMSamhita Alla
LiteralCollection
with multiple literal types? If so, how should the get_literal_type
method be defined since collection_type
can only accept a single LiteralType
? See https://github.com/flyteorg/flytekit/pull/1269/files PR for reference.Ryan Nazareth
11/09/2022, 10:53 AMcryptic
11/12/2022, 5:06 AMRyan Nazareth
11/13/2022, 2:48 AMcryptic
11/14/2022, 8:24 AMRyan Nazareth
11/23/2022, 2:28 PMtests/flytekit/unit/extras/tensorflow/record/test_record.py
, which has tasks to generate TFRecordFile and TFRecordDirectory from tf.train.Example outputs and subsequently deserialise back to tf.data.TFRecordDataset
if required.
The bit thats causing an error is
@task
def consume(dataset: Annotated[TFRecordDatasetV2, TFRecordDatasetConfig(name="testing")]):
which, seems to not be recognising the Annotated TFRecordDatasetV2 type although it should be the expected type in the transformers i have registered here flytekit/extras/tensorflow/record.py.
I presume this is why it is defaulting to FlytePickle as in the traceback (this is only my assumption and based on some other logic i have seen elsewhere in the codebase). Also, my other test tests/flytekit/unit/extras/tensorflow/record/test_transformations.py
which is testing the to_python_value
and to_literal
logic works fine so i feel something else is an issue which I am finding difficult to spot at the moment.
expected_python_type = typing.Annotated[flytekit.types.pickle.pickle.FlytePickle.__class_getitem__.<locals>._SpecificFormatClass, TFRecordDatasetConfig(compression_type=None, buffer_size=None, num_parallel_reads=None, name='testing')]
def to_python_value(self, ctx: FlyteContext, lv: Literal, expected_python_type: Type[T]) -> T:
uri = lv.scalar.blob.uri
# Deserialize the pickle, and return data in the pickle,
# and download pickle file to local first if file is not in the local file systems.
if ctx.file_access.is_remote(uri):
local_path = ctx.file_access.get_random_local_path()
ctx.file_access.get_data(uri, local_path, False)
uri = local_path
with open(uri, "rb") as infile:
> data = cloudpickle.load(infile)
E _pickle.UnpicklingError: invalid load key, '5'.
Ryan Nazareth
11/29/2022, 10:00 PMSamhita Alla