Validating yaml to json conversion

When using yaml to json conversion as it happens when using Terraform’s helm provider with a helm chart’s values.yaml file, it won’t be possible to check and get any useful validation errors, even when your terraform plan is run.

In order to check before running your plan or if you have any errors you can use yq passing as below using the values.yaml file

yq -p yaml -o json values.yaml

Post Process Forwarder – KafkaError “Offset Out of Range (Kubernetes – Sentry – Helm)

Problem

After an upgrade of a self hosted instance of Sentry in Kubernetes with a Helm chart, you are getting the following error and one or more pods are keep failing with:

Post Process Forwarder - KafkaError "Offset Out of Range

Solution

There is a section in Sentry’s documentation here that describes the issue and leads to the comment here

The comment and the steps are using the –bootstrap-server 127.0.0.1:9092 flag which is the one that works.

It is also important to run the command in the group that you have the issue with (ie snuba-events-subscriptions-consumers) to fix this.

So find out the kafka-0 pod in your kubernetes and login to it

kubectl -n sentry exec -it sentry-kafka-0 -- /bin/bash

Get a list of the groups

I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list
snuba-consumers
ingest-consumer
transactions_group
snuba-post-processor
snuba-events-subscriptions-consumers
subscriptions-commit-log-1de9aaa...
snuba-post-processor:sync:880fbbb...
subscriptions-commit-log-b755cccc...
snuba-replacers
query-subscription-consume
r

Run the command in the group you have the issue (snuba-events-subscriptions-consumers)

I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute               

GROUP                          TOPIC                          PARTITION  NEW-OFFSET      
snuba-events-subscriptions-consumers events                         0          4834425   
  

Grafana blank graphs in dashboards after update

Upgrading Grafana (kube-prometheus-stack) from version 23.1 to the latest one currently 44.2.1 results in many Grafana graphs to disappear from the existing dashboards.

Since editing the actual graphs and trying to use ‘Run queries’ does not seem to work, a work around is the following:

  • Edit the blank graph.
  • Add a new query (B) – even empty is fine
  • Use the ‘Run queries’ (it does not matter if you use the button on the original one or in the new query B)
  • Delete the new query B
  • The graph should appear again, so use the ‘Apply’ button on top right.
  • Repeat the process for any additional graphs in the dashboard
  • When you finish ‘Save’ the dashboard
  • The dashboard should be working again

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Problem

You are trying to install the gitlab kubernetes agent (kas) after getting the installation instructions from the Gitlab registration, but you are getting the following error:

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Solution

This is caused by the helm version used, so upgrading the helm version works with no issues.

 helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

so trying to install gives the previous error.

Changing the helm version (ie with asdf)

asdf global helm 3.70


Checking the version

helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}

And running the helm installation again

helm upgrade --install gitlab-kas-agent gitlab/gitlab-agent \

--namespace gitlab-agent --set image.tag=v15.6.0 \

--set config.token=token_from_gitlab-registration \

--set config.kasAddress=wss://kas.domain_name.tld


Release "gitlab-kas-agent" does not exist. Installing it now.
NAME: gitlab-kas-agent
LAST DEPLOYED: Mon Dec  5 13:38:33 2022
NAMESPACE: gitlab-agent
STATUS: deployed
REVISION: 1

TEST SUITE: None

Restic index invalid data returned error

Problem

It is possible after a broken network connection to get the restic error about invalid data returned for the index (maybe when using check).

Solution

You would need to rebuild the index, but using the –read-all-packs flag as described [here](https://forum.restic.net/t/fatal-load-index-xxxxxxxxx-invalid-data-returned/3596/27) which does the rebuild from scratch

restic rebuild-index -r $REPO --read-all-packs

PostgreSQL connection string for Percona PostgreSQL K8S operator

Since the documentation does not contain any information about how you can connect an existing application to the newly created percona pgo cluster, you can use something like the following in your pod postgresql connection string.

postgresql://username:password@cluster1.pgo-perc-production.svc.cluster.local/production

where cluster1.pgo-perc-production.svc.cluster.local points to your newly created cluster and the /production is the database to connect to.

googlecloudsdk.calliope.exceptions.HttpException: ResponseError: code=400, message=Autopilot clusters must be regional clusters.

Problem

Trying to create an auto-cluster either using terraform or gcloud cli, and specifying the region name returns the error that ‘Autopilot clusters must be regional clusters.

So with gcloud this is the command and output

kosmas: (master %)$ gcloud container clusters create-auto test-cluster --region=europe-west6-b
Note: The Pod address range limits the maximum size of the cluster. Please refer to https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr to learn how to optimize IP address allocation.
ERROR: (gcloud.container.clusters.create-auto) ResponseError: code=400, message=Autopilot clusters must be regional clusters.

Solution

Using the actual region name (that can be taken from the list of available zones/regions)

gcloud compute zones list

NAME                       REGION                   STATUS  NEXT_MAINTENANCE  TURNDOWN_DATE
us-east1-b                 us-east1                 UP
us-east1-c                 us-east1                 UP
...
europe-west6-b             europe-west6             UP
...

And using the correct region name (without the b)

gcloud container clusters create-auto test-cluster --region=europe-west6 --verbosity debug

...
Created [https://container.googleapis.com/v1/projects/gitlab-runner-343714/zones/europe-west6/clusters/test-cluster].
...
NAME          LOCATION      MASTER_VERSION   MASTER_IP     MACHINE_TYPE  NODE_VERSION     NUM_NODES  STATUS
test-cluster  europe-west6  1.21.6-gke.1503  xxx.xxx.xxx.xxx  e2-medium     1.21.6-gke.1503  3          RUNNING