Zhiyi Li
11/25/2022, 2:18 PM{"json":{"src":"base.go:73"},"level":"fatal","msg":"caught panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:73|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:73> +0x88\npanic({0x224d580, 0xc0009c8300})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(|ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(>{0x2bd10a0, 0xc00005a018}, {0x1, {0xc000f2e460, 0x5}, {{0x0, 0x0}},
The weird thing is that we tried to use the same version sarama client to connect to the kafka instance in the same cluster, and it works. (Also I was able to success at this step, yet after I redeployed with eventTypes set to a single type it failed and could never get back to normal)
The cloudevent config I used is the following:
cloudevents:
enable: true
kafka:
brokers: kafkaip
version:
version:
- 2
- 2
- 0
- 0
eventsPublisher:
eventTypes:
- all
topicName: workflow-engine-test
type: Kafka
Any clue what could be the problem?Samhita Alla
Zhiyi Li
11/28/2022, 11:54 AMDan Rammer (hamersaw)
12/01/2022, 3:42 PMZhiyi Li
12/02/2022, 2:34 AMDan Rammer (hamersaw)
12/02/2022, 6:17 AMZhiyi Li
12/02/2022, 8:03 AMversion:
verion: []int
Yet the admin accessor expects the config to be
Version:
version:[]int
Some part of the above struct conversion is not working properly and the decode fails…Dan Rammer (hamersaw)
12/02/2022, 3:08 PM[2,2,0,0]
and it was incorrectly parsed as [0,0,0,0]
. Is this correct?
Have you tried updating your config to:
cloudevents:
enable: true
kafka:
brokers: kafkaip
Version:
version:
- 2
- 2
- 0
- 0
eventsPublisher:
eventTypes:
- all
topicName: workflow-engine-test
type: Kafka
I'm wondering if there is a specific reason we didn't annotate the Version
with a JSON name because it's the only way to properly decode the sarama.KafkaVersion
.Version
does not work.version
variable in the sarama.Version
is not exported, I'm not sure we can parse it through the config. It looks like the sarama client exposes versions through it's own global variables (ex. V2_2_0
, ..., etc). We probably need to use those to correctly allow users to configured the client. Does this sound correct to you?Zhiyi Li
12/05/2022, 2:37 AMDan Rammer (hamersaw)
12/05/2022, 2:45 PMKevin Su
12/06/2022, 12:34 AM