Hi! I've only just started with flyte, I have it deployed in k8s cluster, with s3 set up for object storage. When I try to run the example task with

pyflyte run --remote --project  jiri-test --domain development hello_world.py hello_world_wf

it fails with

FlyteSystemException: SYSTEM:Unknown: error=None, cause=<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "failed to create a signed url. Error: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to create a signed url. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see
aws.Config.CredentialsChainVerboseErrors", grpc_status:13}"

What's wrong, problem with s3 connection? I can access it from console (aws s3 ls works), I'm logged in using sso...

https://github.com/flyteorg/flyte/blob/2cd006a3b74edab7d6c51b2c7d1b8a1a4b7732d3/flyteadmin/dataproxy/service.go#L144 Looks like misconfiguration of Flyte Admin (backend) and its access to S3

What kind? I have charts/flyte-binary chart, and this for the backend:

storage:
    metadataContainer: flyte-store-eu-1
    userDataContainer: flyte-store-eu-1
    provider: s3
    providerConfig:
      s3:
        region: "eu-west-1"
        authType: "iam"

but I do have propeller component disabled, could it be an issue?

but I do have propeller component disabled, could it be an issue?

While propeller is not the component that uses the presigned url to upload the compiled code, it's the one running the execution so there's no chance it will work without propeller

it's on-prem kubernetes where I do not have as a user rights to directly create new resources, so I disabled all parts that actively do that (still have those errors in log but at least app is deployed)

I do not have as a user rights to directly create new resources

but the Flyte services would have them via the ClusterRole that gets created, or is that restricted too?

yes, that's also restricted, that's why I have rbac.create desabled too

I think the AWS CLI will use the credentials you have in your profile. Is your deployment on prem but the bucket in AWS? In any case, Flyte will try to get a presigned URL using credentials. For on-prem you'd typically use

accesskey

as the

authType

as shown here

well I have

authType: "iam"

as shown above...

How would that work for a backend service? I don’t see what your personal profile has to do with this tbh. When you use the “iam” authtype i believe the library will just defer to an IAM role attached to your pod via web identity. So if you haven’t done any work to make that happen the credential provider will come up with nothing.

That makes sense, if it's only pod that would be accessing s3 storage. However it's not, is it? I had minio setup before, used authkey values and URL that is only resolvable from kubernetes; but then when I tried to run the workflow, it complained (on my cli) that it cannot reach minio server, of course it could not because it was trying to upload the artifacts from my console, not from the pod. So that makes it confusing

flyte-binary does not but I added support for this with flyte-core: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L646

Nice. No plan to do it for other charts? I guess I can prepare such PR myself, if I find a time

I’m not sure, my company just uses Flyte core since we run it in production so that’s the only one I bothered to update

I think in flyte-binary you can do it using

inlineSecretRef

https://github.com/flyteorg/flyte/blob/e38f32c83af15304f264fa99d8518a2ba153293f/charts/flyte-binary/templates/deployment.yaml#L291-L294 Here an example using it to pass the postgres DB password but the secret contents could be the storage credentials

Thanks for hint, I'll take a look