After an upgrade of a self hosted instance of Sentry in Kubernetes with a Helm chart, you are getting the following error and one or more pods are keep failing with:
Post Process Forwarder - KafkaError "Offset Out of Range
There is a section in Sentry’s documentation here that describes the issue and leads to the comment here
The comment and the steps are using the –bootstrap-server 127.0.0.1:9092 flag which is the one that works.
It is also important to run the command in the group that you have the issue with (ie snuba-events-subscriptions-consumers) to fix this.
So find out the kafka-0 pod in your kubernetes and login to it
kubectl -n sentry exec -it sentry-kafka-0 -- /bin/bash
Get a list of the groups
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --listr
Run the command in the group you have the issue (snuba-events-subscriptions-consumers)
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute
GROUP TOPIC PARTITION NEW-OFFSET
snuba-events-subscriptions-consumers events 0 4834425