brief-leather-11142
11/25/2022, 2:18 PM{"json":{"src":"base.go:73"},"level":"fatal","msg":"caught panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:73|ngithub.com/flyteorg/flyteadmin/pkg/rpc/adminservice.NewAdminServer.func1()\n\t/go/src/github.com/flyteorg/flyteadmin/pkg/rpc/adminservice/base.go:73> +0x88\npanic({0x224d580, 0xc0009c8300})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(|ngithub.com/flyteorg/flyteadmin/pkg/async/cloudevent.NewCloudEventsPublisher(>{0x2bd10a0, 0xc00005a018}, {0x1, {0xc000f2e460, 0x5}, {{0x0, 0x0}},
The weird thing is that we tried to use the same version sarama client to connect to the kafka instance in the same cluster, and it works. (Also I was able to success at this step, yet after I redeployed with eventTypes set to a single type it failed and could never get back to normal)
The cloudevent config I used is the following:
cloudevents:
enable: true
kafka:
brokers: kafkaip
version:
version:
- 2
- 2
- 0
- 0
eventsPublisher:
eventTypes:
- all
topicName: workflow-engine-test
type: Kafka
Any clue what could be the problem?tall-lock-23197
brief-leather-11142
11/28/2022, 11:54 AMhallowed-mouse-14616
12/01/2022, 3:42 PMbrief-leather-11142
12/02/2022, 2:34 AMhallowed-mouse-14616
12/02/2022, 6:17 AMbrief-leather-11142
12/02/2022, 8:03 AMversion:
verion: []int
Yet the admin accessor expects the config to be
Version:
version:[]int
Some part of the above struct conversion is not working properly and the decode fails…brief-leather-11142
12/02/2022, 8:31 AMhallowed-mouse-14616
12/02/2022, 3:08 PM[2,2,0,0]
and it was incorrectly parsed as [0,0,0,0]
. Is this correct?
Have you tried updating your config to:
cloudevents:
enable: true
kafka:
brokers: kafkaip
Version:
version:
- 2
- 2
- 0
- 0
eventsPublisher:
eventTypes:
- all
topicName: workflow-engine-test
type: Kafka
I'm wondering if there is a specific reason we didn't annotate the Version
with a JSON name because it's the only way to properly decode the sarama.KafkaVersion
.hallowed-mouse-14616
12/02/2022, 3:09 PMhallowed-mouse-14616
12/02/2022, 3:18 PMVersion
does not work.hallowed-mouse-14616
12/02/2022, 3:26 PMversion
variable in the sarama.Version
is not exported, I'm not sure we can parse it through the config. It looks like the sarama client exposes versions through it's own global variables (ex. V2_2_0
, ..., etc). We probably need to use those to correctly allow users to configured the client. Does this sound correct to you?brief-leather-11142
12/05/2022, 2:37 AMhallowed-mouse-14616
12/05/2022, 2:45 PMglamorous-carpet-83516
12/06/2022, 12:34 AM