Another question: I’m trying to get the returned StructureDataset from a task to write to a desired destination on s3 by creating the StructureDataset via

StructuredDataset(dataframe=spark_df, uri=<s3a://mybucket/foo/bar/>)

but I get the error:

py4j.protocol.Py4JJavaError: An error occurred while calling o247.parquet.
    : java.nio.file.AccessDeniedException: foo/bar/_temporary/0: PUT 0-byte object  on foo/bar/_temporary/0: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: …; S3 Extended Request ID: …=; Proxy: null), S3 Extended Request ID: …=:AccessDenied
    	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:255)

but if not setting uri, the data is written successfully (to a different bucket). And if I write directly via

spark_df.write.parquet

to

<s3a://mybucket/foo/bar/>

in the task it also works. Could the bucket be dropped somehow from the path? Running the workflow locally and writing to local uri:s also work.