gunzip, k8up, ‘unexpected end of file’

Problem

When using the k8up annotations for a postgresql backup with :

k8up.io/backupcommand: /bin/bash -c 'pg_dumpall --clean | gzip --stdout

do not finish correctly, and as a result the gunzip complains about "unexpected end of file"

Solution

There is a workaround by first saving the file and then sending it to the standard output with

k8up.io/backupcommand: /bin/bash -c 'pg_dumpall --clean | gzip -c > /tmp/backup.gz
      && cat /tmp/backup.gz && rm /tmp/backup.gz'

Thanks Simon Beck for the suggestion https://community.appuio.ch/channel/k8up/thread/j8AcG6ZjgGbQAzth5?msg=xgTanRqqJbBkNj9sd

PostgreSQL upgrading from 9.6 to 15.3 has authentication failed (PG::ConnectionBad)

After upgrading a PostgreSQL instance from version 9.6 to the currently latest version 15.3, your application cannot connect to the server, giving an authentication error even if the user/password used is still the same.

This seems to be the result of a change in the method used in the pg_hba.conf file.

Version 9.6 uses md5 but version 15.3 uses scram-sha-256

Changing this in the file pg_hba.conf file from scram-sha-256 to md5 and reloading the configuration file from inside the database with select pg_reload_conf(); , results in the application being able to connect to the database server again.

Update: It is possible to use the recommended scram-sha-256 hash, by doing the following:

  • Uncomment the line with #password_encryption = scram-sha-256 # scram-sha-256 or md5 in file /var/lib/postgresql/data/pgdata/postgresql.conf
  • reload the configuration in the database select pg_reload_conf();
  • recreate the password with the new encryption production=# alter user user_name with password 'password';

Downloading gitlab database backup from s3

You want to download a copy of your gitlab backup (ie when upgrading to version 7.0.0 as described in [here](https://docs.gitlab.com/charts/installation/database_upgrade.html#steps-for-upgrading-the-bundled-postgresql) ).

Login to the toolbox pod and get the credentials file for s3

cat /etc/gitlab/.s3cfg

Create a new file locally (ie ~/.s3cfg_gitlab) and then download the file locally with

s3cmd get s3://gitlab-backups/database_upgrade_6_gitlab_backup.tar  20230531_gitlab_backup.tar -c ~/.s3cfg_gitlab

Validating yaml to json conversion

When using yaml to json conversion as it happens when using Terraform’s helm provider with a helm chart’s values.yaml file, it won’t be possible to check and get any useful validation errors, even when your terraform plan is run.

In order to check before running your plan or if you have any errors you can use yq passing as below using the values.yaml file

yq -p yaml -o json values.yaml

asdf gcloud ERROR: gcloud failed to load: No module named ‘_sqlite3’

Trying to use a gcloud asdf installation it fails with the error:

gcloud version
ERROR: gcloud failed to load: No module named '_sqlite3'

After installing various components (readline, tk etc) the last one to install to make gcloud work is liblzma-dev (sudo apt install liblzma-dev)

Then gcloud works

gcloud version
Google Cloud SDK 420.0.0
bq 2.0.86
bundled-python3-unix 3.9.16
core 2023.02.24
gcloud-crc32c 1.0.0
gsutil 5.20
Updates are available for some Google Cloud CLI components.  To install them,
please run:
  $ gcloud components update

Post Process Forwarder – KafkaError “Offset Out of Range (Kubernetes – Sentry – Helm)

Problem

After an upgrade of a self hosted instance of Sentry in Kubernetes with a Helm chart, you are getting the following error and one or more pods are keep failing with:

Post Process Forwarder - KafkaError "Offset Out of Range

Solution

There is a section in Sentry’s documentation here that describes the issue and leads to the comment here

The comment and the steps are using the –bootstrap-server 127.0.0.1:9092 flag which is the one that works.

It is also important to run the command in the group that you have the issue with (ie snuba-events-subscriptions-consumers) to fix this.

So find out the kafka-0 pod in your kubernetes and login to it

kubectl -n sentry exec -it sentry-kafka-0 -- /bin/bash

Get a list of the groups

I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list
snuba-consumers
ingest-consumer
transactions_group
snuba-post-processor
snuba-events-subscriptions-consumers
subscriptions-commit-log-1de9aaa...
snuba-post-processor:sync:880fbbb...
subscriptions-commit-log-b755cccc...
snuba-replacers
query-subscription-consume
r

Run the command in the group you have the issue (snuba-events-subscriptions-consumers)

I have no name!@sentry-kafka-0:/$ kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute               

GROUP                          TOPIC                          PARTITION  NEW-OFFSET      
snuba-events-subscriptions-consumers events                         0          4834425