Connection refused when using http monitoring with blackbox_exporter and prometheus

Problem

You are trying to set up http endpoint monitoring with prometheus and blackbox_exporter by specifying something like the following:

...
 job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: 
        - 'localhost:8080'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115
...

but you are getting the error (in prometheus targets) that connection was refused.

Solution

First specify the ip protocol as ip4 if you are not using ip6 like:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: "ip4"
      valid_status_codes: [200]
      method: GET

And then make sure that you use the container IP address in the replacement field if you have started blackbox_exporter as a docker container (tip taken from here), which you can find by looking for the container IP address (docker inspect blackbox_exporter | grep IPAddress).

So your prometheus configuration should look like the following (different replacement IP):

 job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: 
        - 'localhost:8080'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.17.0.3:9115

Docker Cloud moving one node from one account to another

Problem

You have used ‘Bring your own node’ to use a server with one account, but you want to change the account and move the node to the new one.

Solution

Use your old account to terminate the node or login to the server and use the following to stop the running dockercloud-agent:

sudo service dockercloud-agent stop

Go to the Docker Cloud web ui interface, and use the 'Bring your own node'. Copy the token that appears after the sudo -H sh -s on the popup window.

On the server update the token with the one from the new account that you just copied, by running the following:

sudo dockercloud-agent set Token=xxxxx_new_token_xxxx

And now restart the dockercloud-agent on the server:

sudo dockercloud-agent restart

You should now be able to see 'Node xxxx detected' on the Docker Cloud web ui after a couple of minutes.

FATAL -> Failed to fork – Digital Ocean – Debian – Docker

Problem

When you are trying to install a package inside a debian Docker container you get the following error message:

FATAL -> Failed to fork.

Solution

This is an indication of ‘out of memory’ so add a swap partition if it does not exist, on the host with the following:

fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile swap swap sw 0 0" >> /etc/fstab

(Thanks Valent)

Docker – Debian – Kernel panick – Automatic restart

Problem

You have a remote server (Debian) running some Docker containers and you want to make sure that they all restart in the event of a kernel panick.

solution

  • Add an automatic restart after a kernel panick by running:
    sysctl kernel.panic=20
  • Emulate a kernel panick by running the following (SysRq more info here :
    echo c > /proc/sysrq-trigger
  • Check that the containers have restarted again with docker ps (NOTE: the containers should have been enabled to autorestart)

Docker Debian locales installation

Problem

You want to install some default locales in a Docker image, and the suggestion is to use locale-gen to do so, but it does not work as expected.

Solution

In order to be able to use the locale-gen and install the locales you need you will have to uncomment the ones needed from /etc/locale.gen first and then use the locale-gen. Example below:

## Set up locales
## Uncomment the ones we need as locale-gen does not work without that
RUN sed -i '/en_US.UTF-8/s/^#//' /etc/locale.gen
RUN sed -i '/de_DE.UTF-8/s/^#//' /etc/locale.gen
## Now install them and set the default one
RUN locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

Dockercloud-agent error: …Error creating default \”bridge\” network:…

Problem

When trying to use the dockercloud-agent with docker you have the following error in the /var/log/dockercloud/agent.log

level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: cannot create network xxx... (docker0): conflicts with network yyy.... (docker0): networks have same bridge name" 

Solution

If you remove the file var/lib/docker/network/files/local-kv.db as suggested here, then if you look again at the log files, it should be a message that the docker daemon has been started:

016/11/17 10:30:28 Respawning docker daemon
2016/11/17 10:30:28 Starting docker daemon: [/usr/bin/docker daemon -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --userland-proxy=false --tlscert /etc/dockercloud/agent/cert.pem --tlskey /etc/dockercloud/agent/key.pem --tlscacert /etc/dockercloud/agent/ca.pem --tlsverify]
2016/11/17 10:30:28 Docker daemon (PID:4036) has been started

Clearing unused (dangling) docker volumes

Problem

After you start using and building Docker images, there is a case that there is a lot of space used even after deleting images and containers (a few GB maybe).

Solution

To clean out the volumes that are no longer used (ie dangling) use the following:

sudo docker volume ls -qf dangling=true | xargs -r sudo docker volume rm

Error response from daemon: conflict: unable to delete xxx (must be forced) – image is referenced in one or more repositories

Problem

Trying to remove some unused docker images that you no longer use with docker rmi image_id you get the following error message:

Error response from daemon: conflict: unable to delete 75d009de0479 (must be forced) - image is referenced in one or more repositories

Solution

Use the following to delete multiple images (ie ubuntu):

docker images ubuntu | tail -n +2 | awk '{ print $1 ":" $2 }' | xargs sudo docker rmi

Mentioned here