quiet-grass-7164
03/08/2023, 5:11 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f2d93e14fa6164601aad-n0-0] terminated with exit code (137). Reason [OOMKilled]. Message:
{"asctime": "2023-03-08 13:31:31,597", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-mmqveczs because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
.
gray-ocean-62145
03/08/2023, 5:17 PMfrom flytekit import Resources, task
@task(limits=Resources(mem="256Mi")
def your_task(...
freezing-airport-6809
quiet-grass-7164
03/08/2023, 5:28 PMTraceback (most recent call last):
File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/usr/local/lib/python3.10/site-packages/flytekit/tools/fast_registration.py", line 111, in download_distribution
FlyteContextManager.current_context().file_access.get_data(additional_distribution, os.path.join(destination, ""))
File "/usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 456, in get_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://senn-ai-mlops-flyte/xa/flytesnacks/development/MZGUPI5PR6WYBPOECGZDQE2L4A======/scriptmode.tar.gz> to /root/ (recursive=False).
Original exception: Access Denied
freezing-airport-6809
quiet-grass-7164
03/08/2023, 5:36 PMstorage:
metadataContainer: "senn-ai-mlops-flyte"
userDataContainer: "senn-ai-mlops-flyte"
provider: s3
providerConfig:
s3:
region: "eu-central-1"
authType: "iam"
quiet-grass-7164
03/08/2023, 6:14 PMAWS_DEFAULT_REGION : eu-central-1
AWS_METADATA_SERVICE_NUM_ATTEMPTS : 20
AWS_METADATA_SERVICE_TIMEOUT : 5
AWS_REGION : eu-central-1
AWS_ROLE_ARN : xxxxxxx/playground-eks-flyte-user
AWS_STS_REGIONAL_ENDPOINTS : regional
AWS_WEB_IDENTITY_TOKEN_FILE : /var/run/secrets/eks.amazonaws.com/serviceaccount/token
average-finland-92144
03/08/2023, 8:00 PMkubectl annotate serviceaccount -n flyte default <<http://eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/playground-eks-flyte-user|eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/playground-eks-flyte-user>>
In your case, check if you're using a different namespace name instead of flyte
average-finland-92144
03/08/2023, 8:02 PMquiet-grass-7164
03/08/2023, 8:09 PMUNKNOWN::Outputs not generated by task execution
average-finland-92144
03/08/2023, 8:13 PMquiet-grass-7164
03/08/2023, 8:49 PMtall-lock-23197
quiet-grass-7164
03/09/2023, 8:20 AMtall-lock-23197
quiet-grass-7164
03/09/2023, 9:30 AM