acoustic-carpenter-78188
05/21/2026, 6:54 PMPermissionDeniedError: ... HEAD https://s3.us-east-2.amazonaws.com/
union-cloud-dogfood-2-dogfood-fast-registration/.../fast79780202...tar.gz
... 403 Forbidden (empty body)
The runner pod's entrypoint calls flyte.storage.get(s3://...) to download the fast-registration tarball but obbstore couldn't auth.
I also saw this consistently in spark tasks:
"Pod failed. No message received from kubernetes. [spark-kubernetes-driver] terminated with exit code (1). Reason [Error]. Message: ."
I think this is on an IRSA-only pod (no static creds, no profile, no IMDS), obstore has nothing to use → S3 returns 403 / "missing credentials"
According to Claude:
obstore's S3Store(...) constructor calls Rust object_store:awsAmazonS3Builder:from_env(), and that builder only reads static keys: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION, AWS_ENDPOINT_URL.
It does not consult AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ROLE_ARN (which is how IRSA works on EKS). With no static creds present, the builder falls back to InstanceCredentialProvider → IMDS → the EKS node instance role, not the pod's
service-account role. The node role doesn't have S3 access to the fast-registration bucket → 403 with empty body (characteristic of an unsigned-looking request failure).
S3.get_fsspec_kwargs() only wired a credential_provider when both AWS_PROFILE and AWS_CONFIG_FILE were set (the laptop/dev-machine case). On EKS with IRSA, neither is set, so the function returned without a credential provider → obstore went anonymous → 403.For Spark
Spark driver container starts on EKS with an IRSA-annotated service account (spark SA gets eks.amazonaws.com/role-arn=...)
2. EKS pod identity webhook injects AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token into the container env
3. flyte-sdk runs as the entrypoint, needs to read the task's input literal_map from s3://union-cloud-dogfood-2-dogfood/...
4. flyte-sdk uses obstore (Rust object_store crate) for S3
5. obstore's AmazonS3Builder::from_env() consults this credential chain:
- Static env creds (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)
- AWS profile (AWS_PROFILE + AWS_CONFIG_FILE)
- EC2 IMDS (instance metadata)
- NOT AWS_WEB_IDENTITY_TOKEN_FILE (IRSA) — obstore doesn't implement web identity by defaultflyteorg/flyte-sdk
acoustic-carpenter-78188
05/21/2026, 6:54 PMacoustic-carpenter-78188
05/26/2026, 11:02 AM