When using yaml to json conversion as it happens when using Terraform’s helm provider with a helm chart’s values.yaml file, it won’t be possible to check and get any useful validation errors, even when your terraform plan is run.
In order to check before running your plan or if you have any errors you can use yq passing as below using the values.yaml file
yq -p yaml -o json values.yamlThe Helm installation order can be seen here
After an upgrade of a self hosted instance of Sentry in Kubernetes with a Helm chart, you are getting the following error and one or more pods are keep failing with:
Post Process Forwarder - KafkaError "Offset Out of Range
There is a section in Sentry’s documentation here that describes the issue and leads to the comment here
The comment and the steps are using the –bootstrap-server 127.0.0.1:9092 flag which is the one that works.
It is also important to run the command in the group that you have the issue with (ie snuba-events-subscriptions-consumers) to fix this.
So find out the kafka-0 pod in your kubernetes and login to it
kubectl -n sentry exec -it sentry-kafka-0 -- /bin/bash
Get a list of the groups
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --listr
snuba-consumers
ingest-consumer
transactions_group
snuba-post-processor
snuba-events-subscriptions-consumers
subscriptions-commit-log-1de9aaa...
snuba-post-processor:sync:880fbbb...
subscriptions-commit-log-b755cccc...
snuba-replacers
query-subscription-consume
Run the command in the group you have the issue (snuba-events-subscriptions-consumers)
I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute
GROUP TOPIC PARTITION NEW-OFFSET
snuba-events-subscriptions-consumers events 0 4834425
Upgrading Grafana (kube-prometheus-stack) from version 23.1 to the latest one currently 44.2.1 results in many Grafana graphs to disappear from the existing dashboards.
Since editing the actual graphs and trying to use ‘Run queries’ does not seem to work, a work around is the following:
You are trying to install the gitlab kubernetes agent (kas) after getting the installation instructions from the Gitlab registration, but you are getting the following error:
Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action
This is caused by the helm version used, so upgrading the helm version works with no issues.
helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}
so trying to install gives the previous error.
Changing the helm version (ie with asdf)
asdf global helm 3.70
Checking the version
helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}
And running the helm installation again
helm upgrade --install gitlab-kas-agent gitlab/gitlab-agent \
--namespace gitlab-agent --set image.tag=v15.6.0 \
--set config.token=token_from_gitlab-registration \
--set config.kasAddress=wss://kas.domain_name.tld
Release "gitlab-kas-agent" does not exist. Installing it now.
NAME: gitlab-kas-agent
LAST DEPLOYED: Mon Dec 5 13:38:33 2022
NAMESPACE: gitlab-agent
STATUS: deployed
REVISION: 1TEST SUITE: None
To get the notes supplied with a specific helm release (ie might be useful if you have multiple repositories but not sure which one is used as with keycloak), use the following:
helm -n namespace get notes helm_release
It is possible after a broken network connection to get the restic error about invalid data returned for the index (maybe when using check).
You would need to rebuild the index, but using the –read-all-packs flag as described [here](https://forum.restic.net/t/fatal-load-index-xxxxxxxxx-invalid-data-returned/3596/27) which does the rebuild from scratch
restic rebuild-index -r $REPO --read-all-packs
Since the documentation does not contain any information about how you can connect an existing application to the newly created percona pgo cluster, you can use something like the following in your pod postgresql connection string.
postgresql://username:password@cluster1.pgo-perc-production.svc.cluster.local/production
where cluster1.pgo-perc-production.svc.cluster.local points to your newly created cluster and the /production is the database to connect to.
Trying to create an auto-cluster either using terraform or gcloud cli, and specifying the region name returns the error that ‘Autopilot clusters must be regional clusters.
So with gcloud this is the command and output
kosmas: (master %)$ gcloud container clusters create-auto test-cluster --region=europe-west6-b Note: The Pod address range limits the maximum size of the cluster. Please refer to https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr to learn how to optimize IP address allocation. ERROR: (gcloud.container.clusters.create-auto) ResponseError: code=400, message=Autopilot clusters must be regional clusters.
Using the actual region name (that can be taken from the list of available zones/regions)
gcloud compute zones list NAME REGION STATUS NEXT_MAINTENANCE TURNDOWN_DATE us-east1-b us-east1 UP us-east1-c us-east1 UP ... europe-west6-b europe-west6 UP ...
And using the correct region name (without the b)
gcloud container clusters create-auto test-cluster --region=europe-west6 --verbosity debug ... Created [https://container.googleapis.com/v1/projects/gitlab-runner-343714/zones/europe-west6/clusters/test-cluster]. ... NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS test-cluster europe-west6 1.21.6-gke.1503 xxx.xxx.xxx.xxx e2-medium 1.21.6-gke.1503 3 RUNNING
When you are trying to add the contents of the Google Cloud credentials json file as a variable in Terraform Cloud you get the error that it cannot contain new lines. In this case you have to use the jq -c option as in:
cat credentials.json | jq -c