cuddly-jelly-27016
08/03/2025, 12:12 AM__init__() got an unexpected keyword argument 'connection'
Working:
• Code snippet inside docker image
Not working:
• Same docker image when we registered in Flyte.
### Expected behavior
We are doing an athena query by connecting to AWS and we want the result in the pandas df. We are able to see query getting executing in Athena only issue is while reading it from S3.
Expected behaviour : Reading data from s3 and converting to pandas df. This issue is only occurring inside flyte cluster
### Additional context to reproduce
Context: So we are doing an Athena query by connecting to AWS and we want the result in the pandas df. We are able to see query getting executing in Athena only issue is while reading it from S3.
So I separately tried to call s3 bucket or run aws commands using python subprocess. I do have access and able fetch the s3 objects.
But my assumption is when its trying to fetch from Pyathena connection is getting lost in flyte.
Flow:
1. Step1: It calls execute function in Pyathena https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/cursor.py#L121C9-L121C16
2. Step 2: tries to get Athena pandas result set https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/cursor.py#L160
3. Step 3: AthenaPandasResultSet has as_pandas function which internally calls read_csv function and the error is causing there https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/result_set.py#L295
I don't know why the flyte is unable to establish a connection to s3 when we call from Pyathena.
### Screenshots
All the screenshot and sample code is available in https://github.com/jayanthshimoga/flytefit Please use your own AWS access key, Secret key and S3 path.
### Are you sure this issue hasn't been raised already?
• Yes
### Have you read the Code of Conduct?
• Yes
flyteorg/flytecuddly-jelly-27016
08/03/2025, 12:12 AM