The Helm installation order can be seen here
Helm installation order
Reply
Tag Archives: helm
Post Process Forwarder – KafkaError “Offset Out of Range (Kubernetes – Sentry – Helm)
Problem
After an upgrade of a self hosted instance of Sentry in Kubernetes with a Helm chart, you are getting the following error and one or more pods are keep failing with:
Post Process Forwarder - KafkaError "Offset Out of Range
Solution
There is a section in Sentry’s documentation here that describes the issue and leads to the comment here
The comment and the steps are using the –bootstrap-server 127.0.0.1:9092 flag which is the one that works.
It is also important to run the command in the group that you have the issue with (ie snuba-events-subscriptions-consumers) to fix this.
So find out the kafka-0 pod in your kubernetes and login to it
kubectl -n sentry exec -it sentry-kafka-0 -- /bin/bash
Get a list of the groups
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --listr
snuba-consumers
ingest-consumer
transactions_group
snuba-post-processor
snuba-events-subscriptions-consumers
subscriptions-commit-log-1de9aaa...
snuba-post-processor:sync:880fbbb...
subscriptions-commit-log-b755cccc...
snuba-replacers
query-subscription-consume
Run the command in the group you have the issue (snuba-events-subscriptions-consumers)
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute
GROUP TOPIC PARTITION NEW-OFFSET
snuba-events-subscriptions-consumers events 0 4834425
Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action
Problem
You are trying to install the gitlab kubernetes agent (kas) after getting the installation instructions from the Gitlab registration, but you are getting the following error:
Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action
Solution
This is caused by the helm version used, so upgrading the helm version works with no issues.
helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}
so trying to install gives the previous error.
Changing the helm version (ie with asdf)
asdf global helm 3.70
Checking the version
helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}
And running the helm installation again
helm upgrade --install gitlab-kas-agent gitlab/gitlab-agent \
--namespace gitlab-agent --set image.tag=v15.6.0 \
--set config.token=token_from_gitlab-registration \
--set config.kasAddress=wss://kas.domain_name.tld
Release "gitlab-kas-agent" does not exist. Installing it now.
NAME: gitlab-kas-agent
LAST DEPLOYED: Mon Dec 5 13:38:33 2022
NAMESPACE: gitlab-agent
STATUS: deployed
REVISION: 1TEST SUITE: None
Get Helm release notes
To get the notes supplied with a specific helm release (ie might be useful if you have multiple repositories but not sure which one is used as with keycloak), use the following:
helm -n namespace get notes helm_release
Helm list does not produce the full list of deployments
Problem
You are using helm list and there is a time that you don’t get the full list back, after some time.
Solution
By default helm only displays 256 items, so if you are over this default limit you will need to add the –max 0 flag to the command like
helm list --max 0Enable (opt in) Grafana Unified Alerting in your values.yaml
In order to enable the new Grafana Unified alerting, you can add the following to your helm chart’s values.yaml file and upgrade your helm chart with the new values.
grafana.ini:
...
unified_alerting:
enabled: true
alerting:
enabled: false