Hi community, I am facing an error in cluster runs, everything runs successfully, but I cannot see or query the inputs / outputs of certain nodes. When I dig a bit deeper, I find this error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "Received message larger than max (6015882 vs. 4194304)"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:192.168.1.253:30080 {created_time:"2024-05-13T16:07:32.782822331+02:00", grpc_status:8, grpc_message:"Received message larger than max (6015882 vs. 4194304)"}"

Is there a way to set this limit higher and what can cause this issue? Thank you for any insight!

Hey @bored-needle-72209 You can try setting maxMessageSizeBytes Let us know if it helps

Thanks, where can I set it?

it would be in your Helm values file. What chart are you using, flyte-binary or flyte-core?

We are using flyte-binary

ok so that would be on

configuration:
  inline:
    server:
      maxMessageSizeBytes: <insert-value>

Sorry that I didn't reply, thank your for your help, I could solve it so that I can get the data now from the dashboard, but I still can't reach the outputs by FlyteRemote locally on my machine. I tried to put this configurtation into ~/.flyte/config.yaml, but didn't work. Where should I set this configuration?

hey Peter, so to set this option on you local config file it should be like this:

admin:
  endpoint:...
  ...
server:
  grpc:
    maxMessageSizeBytes: <insert-value>

hmm thanks, but still not working, are you referring to the config at ~/.flyte/config.yaml ? I have something like this, but the message size bump has no effect. admin: endpoint: <> insecure: true insecureSkipVerify: true storage: connection: endpoint: <> access-key: <> secret-key: <> server: grpc: maxMessageSizeBytes: 10000000

I guess that's because this is a server side setting, Do you get the same error when trying to fetch outputs with FlyteRemote?

Yes, same error with FlyteRemote