OpenTelemetry docker demo error for otelcol container

Posted on by
Reply

Problem

You are trying to set up the OpenTelemetry docker demo application but the otel container fails with errors similar to the followng:

Error: failed to resolve config: cannot resolve the configuration: cannot retrieve the configuration: unable to read the file file:/etc/otelcol-config.yml: open /etc/otelcol-config.yml: permission denied


Error: failed to resolve config: cannot resolve the configuration: cannot retrieve the configuration: unable to read the file file:/etc/otelcol-config-extras.yml: open /etc/otelcol-config-extras.yml: permission denied

Solution

Change the permissions to the two configuration files in src/otelcollector to 644 and then stop and start the container.

Following the log file (docker logs -f otel-col) should have something like the following instead of the previous errors:

2024-01-10T14:44:29.062Z	info	service@v0.91.0/telemetry.go:86	Setting up own telemetry...
2024-01-10T14:44:29.063Z	info	service@v0.91.0/telemetry.go:203	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2024-01-10T14:44:29.064Z	info	exporter@v0.91.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-01-10T14:44:29.064Z	info	exporter@v0.91.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "debug"}

Create password for prometheus authentication

Posted on by
Reply

To create the password hash that can be used in basic authentication with prometheus you can can use the following:

htpasswd -nBC 10 ""  t -d ':\n'


10 is the bcrypt cost. Usually, the cost used should be between 10 and 12, with current computing power. Increasing this number will likely increase the security of the password at the expense of compute resources.

This can be then used in the configuration file like:

tls_server_config:
  cert_file: prometheus.crt
  key_file: prometheus.key
basic_auth_users:
  user: hhshdhdhdhhdhashedpassword

Note: Taken from the book ‘Prometheus Up & Running second edition’ by Julien Pivotto, Brian Brazil from Oreilly

Key is stored in legacy trusted.gpg keyring

Posted on by
Reply

As described in the post [here](https://itsfoss.com/key-is-stored-in-legacy-trusted-gpg/) you would need to export the key to its own file under the trusted.gpg.d directory.

So for example if the warning is for something like forticlient, first find the key using sudo apt-key gpg list and then taking the last eight characters and removing the space, export the key to its file (ie forticlient.gpg)

sudo apt-key export 5E54716D | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/forticlient.gpg

gunzip, k8up, ‘unexpected end of file’

Posted on by
Reply

Problem

When using the k8up annotations for a postgresql backup with :

k8up.io/backupcommand: /bin/bash -c 'pg_dumpall --clean | gzip --stdout

do not finish correctly, and as a result the gunzip complains about "unexpected end of file"

Solution

There is a workaround by first saving the file and then sending it to the standard output with

k8up.io/backupcommand: /bin/bash -c 'pg_dumpall --clean | gzip -c > /tmp/backup.gz
      && cat /tmp/backup.gz && rm /tmp/backup.gz'

Thanks Simon Beck for the suggestion https://community.appuio.ch/channel/k8up/thread/j8AcG6ZjgGbQAzth5?msg=xgTanRqqJbBkNj9sd

PostgreSQL upgrading from 9.6 to 15.3 has authentication failed (PG::ConnectionBad)

Posted on by
Reply

After upgrading a PostgreSQL instance from version 9.6 to the currently latest version 15.3, your application cannot connect to the server, giving an authentication error even if the user/password used is still the same.

This seems to be the result of a change in the method used in the pg_hba.conf file.

Version 9.6 uses md5 but version 15.3 uses scram-sha-256

Changing this in the file pg_hba.conf file from scram-sha-256 to md5 and reloading the configuration file from inside the database with select pg_reload_conf(); , results in the application being able to connect to the database server again.

Update: It is possible to use the recommended scram-sha-256 hash, by doing the following:

  • Uncomment the line with #password_encryption = scram-sha-256 # scram-sha-256 or md5 in file /var/lib/postgresql/data/pgdata/postgresql.conf
  • reload the configuration in the database select pg_reload_conf();
  • recreate the password with the new encryption production=# alter user user_name with password 'password';

Downloading gitlab database backup from s3

Posted on by
Reply

You want to download a copy of your gitlab backup (ie when upgrading to version 7.0.0 as described in [here](https://docs.gitlab.com/charts/installation/database_upgrade.html#steps-for-upgrading-the-bundled-postgresql) ).

Login to the toolbox pod and get the credentials file for s3 

cat /etc/gitlab/.s3cfg

Create a new file locally (ie ~/.s3cfg_gitlab) and then download the file locally with

s3cmd get s3://gitlab-backups/database_upgrade_6_gitlab_backup.tar  20230531_gitlab_backup.tar -c ~/.s3cfg_gitlab