#5241 [BUG] BigQuery structured dataset shouldn't `append` data when creating
Issue created by
austin362667
### Describe the bug
While creating
data = [
{
'company': 'XYZ pvt ltd',
'location': 'London',
'info': {
'president': 'Rakesh Kapoor',
'contacts': {
'email': '
contact@xyz.com',
'tel': '9876543210'
}
}
},
{
'company': 'ABC pvt ltd',
'location': 'USA',
'info': {
'president': 'Kapoor Rakesh',
'contacts': {
'email': '
contact@abc.com',
'tel': '0123456789'
}
}
}
]
df = pd.json_normalize(data, max_level=0)
bq_sd = StructuredDataset(
dataframe=df,
uri= "
bq://flyte-austin362667-gcp:dataset.nested_type"
)
It appends data every time rather than just overriding it.
### Expected behavior
In above example,
it should only show two rows always.
| Company | Location | President | Email | Telephone |
| ----------- | -------- | ------------- | ----------------------------------------- | ---------- |
| XYZ pvt ltd | London | Rakesh Kapoor | [contact@xyz.com](
contact@xyz.com) | 9876543210 |
| ABC pvt ltd | USA | Kapoor Rakesh | [contact@abc.com](
contact@abc.com) | 0123456789 |
### Additional context to reproduce
No response
### Screenshots
However, we get 8 rows when creating same structured dataset 4 times.
[image](https://private-user-images.githubusercontent.com/35886692/323065854-8a9a6173-2730-45a4-a63b-2a6909244bbb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDgxMzIyMDIsIm5iZiI6MTc0ODEzMTkwMiwicGF0aCI6Ii8zNTg4NjY5Mi8zMjMwNjU4NTQtOGE5YTYxNzMtMjczMC00NWE0LWE2M2ItMmE2OTA5MjQ0YmJiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA1MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNTI1VDAwMTE0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkYjcxMWFiOTJjOGJhYzk0MjE2YmVlYTQ0ZmQ5ZDdjNWUwOTRhYzM4NGM1ZGM2Y2YxYTY5ZDRkYWExZjI5OTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.br5iOGt9wXzlCjLYpWOK58Fp0xZGxVEPRd5dlrtKfPw)
### Are you sure this issue hasn't been raised already?
• Yes
### Have you read the Code of Conduct?
• Yes
flyteorg/flyte