OpenTelemetry docker demo error for otelcol container

Posted on 10 January 2024 by kosmas

Problem

You are trying to set up the OpenTelemetry docker demo application but the otel container fails with errors similar to the followng:

Error: failed to resolve config: cannot resolve the configuration: cannot retrieve the configuration: unable to read the file file:/etc/otelcol-config.yml: open /etc/otelcol-config.yml: permission denied

Error: failed to resolve config: cannot resolve the configuration: cannot retrieve the configuration: unable to read the file file:/etc/otelcol-config-extras.yml: open /etc/otelcol-config-extras.yml: permission denied

Solution

Change the permissions to the two configuration files in src/otelcollector to 644 and then stop and start the container.

Following the log file (docker logs -f otel-col) should have something like the following instead of the previous errors:

2024-01-10T14:44:29.062Z	info	service@v0.91.0/telemetry.go:86	Setting up own telemetry...
2024-01-10T14:44:29.063Z	info	service@v0.91.0/telemetry.go:203	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2024-01-10T14:44:29.064Z	info	exporter@v0.91.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-01-10T14:44:29.064Z	info	exporter@v0.91.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "debug"}

client version 1.43 is too new. Maximum supported API version is 1.41

Posted on 5 September 2023 by kosmas

Problem

Trying to use the Docker API to create/stop/delete a container gives the error

client version 1.43 is too new. Maximum supported API version is 1.41

Solution

Use the following

export DOCKER_API_VERSION=1.41

as described here

docker rmi image_name V docker image rm image_name

Posted on 8 February 2021 by kosmas

There is a newer way of using the command, as it is also the case in a few other docker commands.

Even though both the command can be used to remove a docker image, the command using docker image rm is the most recent way of using it.

Failed to start docker.service: Unit is masked

Posted on 27 January 2021 by kosmas

Problem

Trying to start the docker service after some upgrades fails with the following message:

Failed to start docker.service: Unit is masked.

Solution

It turns out that after upgrading or more specifically removing and then upgrading the docker installation in ubuntu (in this particular case in raspberry 4 with Ubuntu 20.04 installed), results in this error.

A search brings up the following:

https://forums.docker.com/t/failed-to-start-docker-service-unit-is-masked/67413

and from that the following bug post:

https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1844894

So the solution is to run the following to be able to start the docker service (described in the first link above):

sudo systemctl unmask docker
sudo systemctl start docker

Get the host IP address from inside a docker container

Posted on 23 April 2019 by kosmas

Problem

You wold like to know the IP Address of your host from inside a running docker container.

Solution

Run the following inside the container:

route | awk '/^default/ { print $2 }'

Cannot create container for service xxx: invalid mode: /path/to/volume/

Posted on 22 October 2018 by kosmas

Problem

You are trying to use docker-compose to get some services up with docker but you see an error like the following:

Cannot create container for service xxx: invalid mode: /path/to/volume/

Solution

This is more than likely caused by a typo in your yml file, so go back and check carefully for any typos and correct them.

ERROR: Couldn’t connect to Docker daemon at http+docker://localunixsocket – is it running?

Posted on 17 October 2018 by kosmas

Problem

Using docker-compose up (or build), displays the following error message (even though the same command used to work previously):

ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?

Solution

This is quite possible caused by permissions issue, as one of the folders files that docker is trying to use is owned by a different user/group from the one trying to use the docker-compose commands. Try to find the file/folder with the different permissions and change it to your user name and group, or use change the files by using something like:

chown -R me:me .

Find the IP Address of a running Docker container

Posted on 4 September 2018 by kosmas

Problem

You would like to know the IP Address of a running Docker container.

Solution

Use the following to find out the IP Address by replacing the container_name with the actual container name:

docker inspect -f '{{ .NetworkSettings.IPAddress }}' container_name

Connection refused when using http monitoring with blackbox_exporter and prometheus

Posted on 14 June 2018 by kosmas

Problem

You are trying to set up http endpoint monitoring with prometheus and blackbox_exporter by specifying something like the following:

...
 job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: 
        - 'localhost:8080'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115
...

but you are getting the error (in prometheus targets) that connection was refused.

Solution

First specify the ip protocol as ip4 if you are not using ip6 like:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: "ip4"
      valid_status_codes: [200]
      method: GET

And then make sure that you use the container IP address in the replacement field if you have started blackbox_exporter as a docker container (tip taken from here), which you can find by looking for the container IP address (docker inspect blackbox_exporter | grep IPAddress).

So your prometheus configuration should look like the following (different replacement IP):

 job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: 
        - 'localhost:8080'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.17.0.3:9115

FATAL -> Failed to fork – Digital Ocean – Debian – Docker

Posted on 3 April 2017 by kosmas

Problem

When you are trying to install a package inside a debian Docker container you get the following error message:

FATAL -> Failed to fork.

Solution

This is an indication of ‘out of memory’ so add a swap partition if it does not exist, on the host with the following:

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile swap swap sw 0 0" >> /etc/fstab

(Thanks Valent)

42

Web notebook about thoughts and discoveries in DevOps, SRE, Software Engineering….

Category Archives: Docker

OpenTelemetry docker demo error for otelcol container

Problem

Solution

client version 1.43 is too new. Maximum supported API version is 1.41

Problem

Solution

docker rmi image_name V docker image rm image_name

Failed to start docker.service: Unit is masked

Problem

Solution

Get the host IP address from inside a docker container

Problem

Solution

Cannot create container for service xxx: invalid mode: /path/to/volume/

ERROR: Couldn’t connect to Docker daemon at http+docker://localunixsocket – is it running?

Find the IP Address of a running Docker container

Connection refused when using http monitoring with blackbox_exporter and prometheus

FATAL -> Failed to fork – Digital Ocean – Debian – Docker