<#4855 [BUG] Circular Dependency issue in Flyte - ...
# flyte-github
c
#4855 [BUG] Circular Dependency issue in Flyte - pyathena Issue created by jayanthshimoga ### Describe the bug Circular Dependency issue in Flyte. We are getting the below error when we are using pandascursor to get the AWS Athena query result in the flyte workflow, which is only occurring when we use PyAthena >3.0.10. However below python code works fine with other application but doesn't work inside flyte. Python code works fine inside docker image, when the same image is registered to flyte cluster the code is breaking.
__init__() got an unexpected keyword argument 'connection'
Working: • Code snippet inside docker image Not working: • Same docker image when we registered in Flyte. ### Expected behavior We are doing an athena query by connecting to AWS and we want the result in the pandas df. We are able to see query getting executing in Athena only issue is while reading it from S3. Expected behaviour : Reading data from s3 and converting to pandas df. This issue is only occurring inside flyte cluster ### Additional context to reproduce Context: So we are doing an Athena query by connecting to AWS and we want the result in the pandas df. We are able to see query getting executing in Athena only issue is while reading it from S3. So I separately tried to call s3 bucket or run aws commands using python subprocess. I do have access and able fetch the s3 objects. But my assumption is when its trying to fetch from Pyathena connection is getting lost in flyte. Flow: 1. Step1: It calls execute function in Pyathena https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/cursor.py#L121C9-L121C16 2. Step 2: tries to get Athena pandas result set https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/cursor.py#L160 3. Step 3: AthenaPandasResultSet has as_pandas function which internally calls read_csv function and the error is causing there https://github.com/laughingman7743/PyAthena/blob/2d88c3e6172295880d011a5f151bc273b9d577ad/pyathena/pandas/result_set.py#L295 I don't know why the flyte is unable to establish a connection to s3 when we call from Pyathena. ### Screenshots All the screenshot and sample code is available in https://github.com/jayanthshimoga/flytefit Please use your own AWS access key, Secret key and S3 path. ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte