faint-smartphone-23356
04/10/2023, 9:22 PMthankful-minister-83577
faint-smartphone-23356
04/10/2023, 9:38 PMthankful-minister-83577
thankful-minister-83577
faint-smartphone-23356
04/10/2023, 9:56 PMthankful-minister-83577
faint-smartphone-23356
04/10/2023, 9:57 PMthankful-minister-83577
thankful-minister-83577
faint-smartphone-23356
04/10/2023, 9:58 PMthankful-minister-83577
faint-smartphone-23356
04/10/2023, 9:58 PMfaint-smartphone-23356
04/10/2023, 10:03 PMthankful-minister-83577
faint-smartphone-23356
04/10/2023, 10:37 PMambitious-spoon-47823
04/13/2023, 1:16 PMself.remote = FlyteRemote(
config=Config.for_endpoint(
endpoint=settings.flyte_endpoint,
insecure=settings.flyte_insecure_connection,
data_config=DataConfig(
s3=S3Config(
access_key_id=settings.flyte_data_s3_access_key,
secret_access_key=settings.flyte_data_s3_secret_key,
)
),
),
default_project=settings.flyte_project,
default_domain=settings.flyte_domain,
data_upload_location=f"s3://{settings.flyte_data_s3_bucket_name}/data",
)
Before the bump in 1.5.0 (we were in 1.4.1), everything works as expected. After the bump, we got that issue:
FlyteAssertion('Failed to put data from /tmp/xxxx_cfe6785a-23aa-414a-a03a-e3eec11fb992_h4ibyrj_.zip to s3://bucket/data/xxxxxxxxxx/xxxx_cfe6785a-23aa-414a-a03a-e3eec11fb992_h4ibyrj_.zip (recursive=False).\n\nOriginal exception: Access Denied')Maybe that will help for the troubleshooting ?
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
inputs.pb
and outputs.pb
files? Or is it only ever the off-loaded data types (like flytefile, structureddataset, etc.)?faint-smartphone-23356
04/13/2023, 7:45 PM<1.5.0
was quick enough for us to do that it sort of stalled out the triaging process - but I will get to this in the next few days.thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
faint-smartphone-23356
04/13/2023, 8:54 PMthankful-minister-83577
pip show boto3
(if that’s installed i mean)thankful-minister-83577
pip freeze
from within the container or your virtualenv (redact anything sensitive)ambitious-spoon-47823
04/14/2023, 7:00 AMadal==1.2.7
adlfs==2023.1.0
aiobotocore==2.4.2
aiohttp==3.8.4
aioitertools==0.11.0
aioresponses==0.7.4
aiosignal==1.3.1
alembic==1.10.3
anyio==3.6.2
arrow==1.2.3
async-timeout==4.0.2
asyncache==0.3.1
asyncpg==0.27.0
attrs==22.2.0
awscli==1.27.111
azure-core==1.26.4
azure-datalake-store==0.0.52
azure-identity==1.12.0
azure-storage-blob==12.15.0
binaryornot==0.4.4
black==23.3.0
botocore==1.29.111
cachetools==5.3.0
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
chardet==5.1.0
charset-normalizer==3.1.0
click==8.1.3
cloudpickle==2.2.1
colorama==0.4.4
cookiecutter==2.1.1
coverage==7.2.3
croniter==1.3.14
cryptography==40.0.1
dataclasses-json==0.5.7
decorator==5.1.1
Deprecated==1.2.13
diskcache==5.5.1
distlib==0.3.6
docker==6.0.1
docker-image-py==0.1.12
docstring-parser==0.15
docutils==0.16
execnet==1.9.0
fastapi==0.95.0
filelock==3.11.0
flyteidl==1.3.17
flytekit==1.5.0
frozendict==2.3.7
frozenlist==1.3.3
fsspec==2023.4.0
gcsfs==2023.4.0
gitdb==4.0.10
GitPython==3.1.31
google-api-core==2.11.0
google-auth==2.17.2
google-auth-oauthlib==1.0.0
google-cloud-core==2.3.2
google-cloud-storage==2.8.0
google-crc32c==1.5.0
google-resumable-media==2.4.1
googleapis-common-protos==1.59.0
graphql-core==3.2.3
greenlet==2.0.2
grpcio==1.53.0
grpcio-status==1.53.0
h11==0.14.0
httpcore==0.17.0
httptools==0.5.0
httpx==0.24.0
identify==2.5.22
idna==3.4
importlib-metadata==6.3.0
iniconfig==2.0.0
isodate==0.6.1
jaraco.classes==3.2.3
jeepney==0.8.0
Jinja2==3.1.2
jinja2-time==0.2.0
jmespath==1.0.1
joblib==1.2.0
keyring==23.13.1
kubernetes==26.1.0
Mako==1.2.4
MarkupSafe==2.1.2
marshmallow==3.19.0
marshmallow-enum==1.5.1
marshmallow-jsonschema==0.13.0
more-itertools==9.1.0
msal==1.21.0
msal-extensions==1.0.0
multidict==6.0.4
mypy-extensions==1.0.0
natsort==8.3.1
newrelic==8.8.0
newrelic-telemetry-sdk==0.4.3
nodeenv==1.7.0
numpy==1.24.2
oauthlib==3.2.2
packaging==23.1
pandas==1.5.3
pathspec==0.11.1
platformdirs==3.2.0
pluggy==1.0.0
portalocker==2.7.0
pre-commit==3.2.2
protobuf==4.22.1
protoc-gen-swagger==0.1.0
py==1.11.0
pyarrow==10.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydantic==1.10.7
PyJWT==2.6.0
pyOpenSSL==23.1.1
pytest==7.3.0
pytest-asyncio==0.21.0
pytest-cov==4.0.0
pytest-mock==3.10.0
pytest-xdist==3.2.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-json-logger==2.0.7
python-multipart==0.0.6
python-slugify==8.0.1
pytimeparse==1.1.8
pytz==2023.3
PyYAML==5.4.1
regex==2023.3.23
requests==2.28.2
requests-oauthlib==1.3.1
responses==0.23.1
retry==0.9.2
rfc3986==1.5.0
rsa==4.7.2
s3fs==0.4.2
s3transfer==0.6.0
SecretStorage==3.3.3
six==1.16.0
smmap==5.0.0
sniffio==1.3.0
sortedcontainers==2.4.0
SQLAlchemy==2.0.9
starlette==0.26.1
starlette-context==0.3.6
statsd==3.3.0
strawberry-graphql==0.171.1
tenacity==8.2.2
text-unidecode==1.3
types-PyYAML==6.0.12.9
typing-inspect==0.8.0
typing_extensions==4.5.0
urllib3==1.26.15
uvicorn==0.21.1
uvloop==0.17.0
virtualenv==20.21.0
watchfiles==0.19.0
websocket-client==1.5.1
websockets==11.0.1
wrapt==1.15.0
yarl==1.8.2
zipp==3.15.0
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
faint-smartphone-23356
04/14/2023, 9:27 PMfaint-smartphone-23356
04/14/2023, 9:28 PMfaint-smartphone-23356
04/14/2023, 9:31 PMfaint-smartphone-23356
04/14/2023, 9:32 PMthankful-minister-83577
thankful-minister-83577
ambitious-spoon-47823
04/15/2023, 8:57 AM{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
To:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": [
"arn:aws:s3:::bucket",
"arn:aws:s3:::bucket/*"
]
}
]
}
faint-smartphone-23356
04/15/2023, 7:36 PM{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::bucket",
"arn:aws:s3:::bucket/*"
]
},
(I've switched the actual bucket name due to paranoia)faint-smartphone-23356
04/18/2023, 4:38 PMfaint-smartphone-23356
04/18/2023, 4:41 PMthankful-minister-83577
thankful-minister-83577
faint-smartphone-23356
04/18/2023, 5:31 PMfaint-smartphone-23356
04/24/2023, 3:09 PMflytekit
and flytekitplugins-pods
= <1.5.0
.
I'm going to roll forward to 1.5.0 again, assuming we'll hit the issue again in the next couple of days while I'll be around to triage.thankful-minister-83577
millions-table-8574
05/10/2023, 7:27 AMmillions-table-8574
05/10/2023, 7:36 AMflytekit==1.5.0
. The issue was “fixed” by rolling back to the previous image (running 1.4.2
)faint-smartphone-23356
05/10/2023, 1:27 PMfaint-smartphone-23356
05/10/2023, 1:30 PMfaint-smartphone-23356
05/10/2023, 1:30 PM