plain-carpenter-67621
06/27/2022, 9:11 PMPod failed. No message received from kubernetes.
[atg6lnllwptbr4j6thwc-n0-0] terminated with exit code (137). Reason [Error]. Message:
.
the flyte cluster is local sandbox clusterplain-carpenter-67621
06/28/2022, 2:03 AMLimits:
cpu: 2
memory: 1Gi
Requests:
cpu: 1
memory: 1Gi
plain-carpenter-67621
06/28/2022, 2:04 AM1Gi
the max limit i can set?tall-lock-23197
kubectl describe po <pod-name> -n flytesnacks-development
?plain-carpenter-67621
06/28/2022, 4:40 AMacceptable-policeman-57188
acceptable-policeman-57188
plain-carpenter-67621
06/28/2022, 3:37 PM@task(requests=Resources(cpu="1", mem="900Mi"), limits=Resources(mem="1G", cpu="2"))
plain-carpenter-67621
06/28/2022, 3:37 PMacceptable-policeman-57188
plain-carpenter-67621
06/28/2022, 3:41 PMplain-carpenter-67621
06/28/2022, 3:41 PMacceptable-policeman-57188
acceptable-policeman-57188
plain-carpenter-67621
06/28/2022, 3:43 PM20Mi
that seems low
"task_resource_defaults.yaml": "task_resources:
defaults:
cpu: 100m
memory: 500Mi
storage: 500Mi
limits:
cpu: 2
gpu: 1
memory: 1Gi
storage: 20Mi
"
plain-carpenter-67621
06/28/2022, 3:45 PMacceptable-policeman-57188
acceptable-policeman-57188
plain-carpenter-67621
06/28/2022, 3:48 PM@task(requests=Resources(cpu="1", mem="900Mi"), limits=Resources(mem="1G", cpu="2"))
def split_traintest_dataset(
dataset: FlyteFile[typing.TypeVar("parquet")], seed: int, test_split_ratio: float
) -> Tuple[
FlyteSchema[NYC_FEATURE_COLUMNS],
FlyteSchema[NYC_FEATURE_COLUMNS],
FlyteSchema[NYC_CLASSES_COLUMNS],
FlyteSchema[NYC_CLASSES_COLUMNS],
]:
"""
Retrieves the training dataset from the given blob location and then splits it using the split ratio and returns the result
"""
column_names = [k for k in NYC_DATASET_COLUMNS.keys()]
try:
df = pd.read_parquet("workflows/yellow_tripdata_2022-01.parquet")
clean_df = preprocess(df)
except Exception as err:
print(err)
# Select all features
x = clean_df[column_names[:-1]]
# Select only the classes
y = clean_df[[column_names[-1]]]
# split data into train and test sets
return train_test_split(x, y, test_size=test_split_ratio, random_state=seed)
acceptable-policeman-57188
plain-carpenter-67621
06/28/2022, 4:07 PMthankful-minister-83577
thankful-minister-83577
-rw-r--r--@ 1 ytong staff 2.5G Mar 24 2021 yellow_tripdata_2010-01.csv
thankful-minister-83577
plain-carpenter-67621
06/28/2022, 5:32 PM@task(requests=Resources(cpu="1", mem="2G", storage="500Mi"), limits=Resources(mem="5G", cpu="2", storage="500Mi"))
the pod crashes.. and i am using a single file yellow_tripdata_2022-01.parquet
which is ~35Mbtall-lock-23197
ephemeral_storage
to say, 500Mi
? Not sure if that’d solve your problem, though.plain-carpenter-67621
06/30/2022, 3:11 PM